We thank reviewer for the feedback and the comments about importance of our topic, the suitability of our evaluation framework, the effectiveness of our approach and the diversity of our test settings. We address the reviewer's comments below.

Computational Limitations of Existing Methods.
Even if 50 frames fit in memory, the real barrier is computational complexity. During training, our SSM updates a fixed-size state once per frame, so compute and activations grow linearly with sequence length $T$. A Transformer-based diffusion must still form full self-attention, which is quadratic $O(T^{2})$ in compute and memory. At inference the gap remains: the SSM’s per-step cost is constant, whereas even KV-cached attention or large-receptive-field CNNs scale linearly with the window. In other words, the SSM can process an increasing sequence in a streaming manner with constant latency. Thus a 50-frame cache alleviates memory but not the quadratic compute bottleneck, and long contexts are typically truncated, forfeiting long-range information.

Long Context Tuning for Video Generation (Guo et al.) trains with full attention over an expanded window (O(N²)), and at inference the per-step cost still scales with the window despite KV cache. We instead compress history into a fixed-size SSM state, enabling linear complexity and constant latency (even with an LCT decoder). Moreover, LCT’s asynchronous shot generation is ill-suited for interactive environments with per-frame actions.

Furthermore, in contrast to methods which finetune/train new models to extend the context, we show that our method is able to extend to much longer sequences than trained without finetuning. We refer to Sup.Mat. Sect. 2.1 where we generalize from context size 16 to context size 50, and an experiment that can be seen in the response of Reviewer dCL402, where we generalize from context size 50 to up to 150.

SSM/Diffusion Training Separation. We confirm that our training consists of two stages: 1) Training the SSM (State Space World Model) 2) Load the weights of the SSM, freeze them, and train the Generative Branch of StateSpaceDiffuser.

Joint (end-to-end) training - from scratch or finetuned - underperformed. Training separately worked best. The two branches learn on different time scales: the SSM needs many sequence-level iterations to stabilize long-horizon state, while the diffusion model quickly fits a local denoising loss and ignores a still-adapting SSM. Finetuning didn’t help: diffusion gradients disrupt the SSM and the decoder reverts to immediate image cues. Freezing the SSM prevents drift and lets the decoder use long-term context. Importantly, the diffusion model does not just "render the SSM’s output": in Sup.Mat. Sec. 3.6 Fig. 11, an ablation shows the SSM alone produces low-quality frames with the right content, while StateSpaceDiffuser—given the same SSM features - produces visibly higher-quality, context-aware results, not a copy of the SSM output. Thus, the diffusion model isn’t bottlenecked by SSM visual quality.

The reviewer is concerned if performing the training separately could be a scalability issue. We view the modular training as a scalability advantage, not a limitation. It decouples learning an efficient, fixed-size memory for long-range semantics (the SSM) from high-fidelity rendering (the diffusion decoder). This lets us upgrade either module without retraining the other (e.g., drop-in a stronger decoder, or train an SSM on longer sequences or more data), making it possible to focus on larger scale training on a single component at a time. As we discuss in the next section, we believe the scalability to more complex environments can improve with larger-scale models - decoupling training does not affect the scalability of the models.

Scalability. We agree that MiniGrid is a controlled setting. This is why we also evaluate on CSGO, which exhibits high-entropy first-person motion, long occlusions/re-appearances, and large viewpoint changes - failure modes typical of real video. Our goal is long-horizon content consistency, which goes beyond maximizing single-frame sharpness. The SSM state is a compact summary by design, while a diffusion decoder recovers visual detail each frame. We alotted most of our limited model compute budget for a lightweight diffusion model, complementing it with a cost-effective SSM. Under a larger compute budget, scaling up the SSM (number of parameters, state size, heads) and the diffusion model (training iterations, more parameters, newer model) is the most straightforward way to improve the visual quality and the capacity of the context preservation, including for more complex environments. For completeness, the final supplement will also show random rollouts (non forward-backward), clarifying that the low quality of the baseline results is most pronounced in the hardest, long-context cases.

Limitations. In Sup.Mat. Sect. 3.2 we have discussed our limitations and attributed them: (1) artifacts after strong action, attributed to the diffusion baseline; (2) decay of context details, attributed to the lossy SSM state (potentially scaling up the SSM can help). As CSGO is a pre-collected dataset of human play, it is difficult to separate clean subsets of failure modes in order to collect failure rate statistics. We can however comment that the decay seems to increase with longer-range dependencies. We can also demonstrate this decay on the predictions of MiniGrid context generalization up to context size 150, which is in contrast to the stable performance for context size up to 50 shown on Fig. 7(b). We show the average PSNR in ranges of 5 for the extended prediction:

Contribution Importance. We agree that, at a high level, leveraging longer history should help. Our contribution is to make this practically effective for diffusion world models: a compact SSM state that carries long-horizon information with constant per-frame latency/memory, an architecture that feeds that information to a diffusion model, and a training scheme that allows for the diffusion model to make use of this information. Naively training end-to-end led to conditioning under-use. While others have used fixed image caches and explicit representations like pointclouds, this is the first work that explores exploiting a state space model for the benefit of a diffusion world model and makes it possible in practice. We consider this a significant contribution. We will improve our methodology and introduction sections to include these points more clearly.

We also contribute a forward–backward evaluation with sufficient motion to test long-horizon consistency. Under this protocol, diffusion-only baselines often predict incorrect content, whereas our model reinstates previously seen content - showing the benefit is not just using more history, but how it is summarized and used under realistic compute. We further observe (see Sup.Mat. Sect. 3.2, Sect 3.6, context generalization in this rebuttal) robustness to large motion, generalization to much longer sequences, and that diffusion renders higher-quality frames later in the rollout than the SSM while still relying on SSM features.

Evaluation Clarifications. We justify the split in Sup.Mat. L111-119. MiniGrid is an environment where an action results in a fixed amount of motion over 1 step. In MiniGrid, each action induces a fixed 1-step motion, so single-frame prediction cleanly isolates context recall from rendering quality. In CSGO, motion varies and unfolds over many frames; one-step prediction mostly probes short-context and both methods look strong. However, multi-step prediction highlights long-horizon consistency. Our model preserves context while the baseline hallucinates, so we adopt multi-step evaluation for CSGO and include MiniGrid multi-step qualitatives in Sup.Mat. Fig. 4.

In CSGO multi-step rollouts accumulate camera/viewpoint drift: even with correct actions, slight motion mismatches de-align later frames (see Sup.Mat.). Content often matches while viewpoint/details differ. Thus PSNR/LPIPS fail to reflect content similarity, which we show experimentally in Sup.Mat. L127–137, Fig. 8. This is a known issue - see High-Resolution Image Synthesis with Latent Diffusion Models (p. 8); Image Super-Resolution via Iterative Refinement (p. 2). From Latent Diffusion: "simple image regression model achieves the highest PSNR and SSIM scores; however these metrics do not align well with human perception and favor blurriness over imperfectly aligned high frequency details." . Following common practice (EmuVideo/Stable Diffusion/Latent Diffusion), we therefore include human preference evaluations.

The Simple MiniGrid setup is an easily reproducible minimal example designed entirely to showcase the the problem and our approach. Even on a minimal color-recall task (two actions, almost no visual complexity, tiny dataset prone to overfitting), the diffusion baseline, limited by its context window, fails. Processing the long history with an SSM lets our model reliably recover the correct color. We then scale difficulty in MiniGrid and CSGO, showing that long-context recall persists as environment complexity rises.

Writing Suggestions. We thank the reviewer for the detailed notes. We will add the requested clarifications on diffusion world model limitations (with the suggested citation), include a Data section figure illustrating MiniGrid and CSGO with example actions, and fix the noted typographical and caption issues.

1. Could you provide an answer to question 3, about line 247 in the paper: "We train and test on the same set of 34 samples, allowing for overfitting."? I'm not quite sure why this is needed/whether it's a limitation of the model.
2. Thanks for the clarification about end to end training. But I still don't see what the Diffusion model is doing, if the SSM is frozen during training. Isn't it simply learning a mapping from the SSM features to the image space? Is it not right to say that we expect the SSM features to implicitly represent everything pertaining to both scene representation and dynamics and the diffusion model is just operating on those features? Or to put it in another way, what exactly is the diffusion model doing that is not already done implicitly by the SSM?
3. How does this method relate to the diffusion forcing approach used in the recent paper on "Long-Context State-Space Video World Models", (Po, et.al.)?
4. Would the authors agree that scaling the SSM training to large scale data can be challenging? If not, could you provide more discussion on why you think this is straightforward? The Po, et.al paper I shared above seems to find that scaling SSMs is hard because of the fixed dimensionality of the latent state, and specialized architectural advances are needed to get them to work. So, I don't think it's right to say that the performance of your model is not limited by the SSM, given that joint training is not possible. I think that's a significant limitation, but I'm happy to have some more discussion on it. I think Reviewer VVHr has also raised a similar question.

Thank you for the feedback. We will make sure to edit the Methodology section to express more clearly the purpose of the two branches.

Positioning among other works in the field

Upon the reviewer suggestion, we plan to update our related work to more clearly position our work among others by including the works and suggestions from all reviews/discussions (incl. other reviewers) in this rebuttal. In addition, we offer some extra insights on our positioning below:

In our paper we have tied our work to diffusion world models and state space models by comparing with DIAMOND - a strong diffusion world model for smaller scale compute; and State Space World Model - a state space model adapted as a world model.

In terms of grounding, we consider the closest work in literature to be S4WM - a model we discuss in main text in Line 39, Lines 126-128 (reference duplication will be fixed). It is a method based on the older S4 state space model that shows promise that SSMs preserve long context in video. Their model is entirely built on an SSM and tested entirely with an agent (no visual fidelity), on simple setups (remember the colors on a single line, or the colors of the walls in a 3D environment with a fixed layout). We were initially excited to compare our independent State Space World Model (our SSM-only model), with this method, and even try it out as our Long-Context Branch. However S4WM's code is not available and the authors did not respond to us. Therefore, we instead performed an ablation study with S4 in the main paper, showing that our State Space World Model, with the Mamba backbone performs better. While this is not a direct comparison with S4WM, the use of S4 bears similarity.

On Lines 116-120 in the main paper we have shortly discussed other recent world model long context methods available at the time, and the relation to our work. To discuss a bit further, those methods are cache-based(storing input images, image representations, etc.) - to handle progressively increasing sequence of high-dimensional data, the cache has to explicitly select/delete sequence elements. Our method differs by forming a compressed representation of the history inside an SSM, capable of keeping information from each frame while throwing away the parts not useful for the current or future states. On Lines 129-134 we position ourselves among other Transformer/Diffusion + SSM methods (which we also plan to extend with discussion).

Limitations and Scalability Discussion

Our results confirm the method’s value under our compute budget. However, like the reviewer, we see the practical value of scaling up the model in future work for more complex environments, under a larger compute budget (Line 125 Sup.Mat.). We will include a section in Sup.Mat. that makes explicit our current model's limitations under the scale we have and instructions on different directions to scale up the model, as discussed in previous comments - by directly scaling model parameters (available in our specific architecture), and other specialized approaches, like parallelizing SSMs (Po et. al.). To be more explicit - in the main paper text we will include comments in the experiment discussions with references to that section. To further improve transparency about the model's limitations, we will include a clear discussion in the main paper on limitations and strengths with the main points already made in Sup.Mat as well as in this rebuttal, referring to the respective Sup.Mat. section for more details.

We believe to have addressed all the questions - relation to prior work, and limitations/extensions discussion. Thank you for the constructive suggestions. We would appreciate your comment and assessment.

Thanks for the clarifications. That addresses all my concerns. One thing I would also suggest you is to move the Sup. Mat. Fig. 11 and the associated discussion to the main paper because I think that makes it easier to understand the method, thanks!

I will improve my rating.

We thank the reviewer for their invaluable feedback and for acknowledging the importance of our topic, the comprehensiveness of our evaluation, and the clarity of our writing. We address the reviewer's comments below.

Computational Cost. In inference, the state space model maintains a state in a streaming fashion, which allows for live processing of a growing sequence. At every time step of the sequence, the state is updated, used to generate the output, and kept for the next step. As we only need to keep the updated state, the latency remains constant - memory complexity and computational complexity per step remain the same and is not dependent on the sequence size. In contrast, transformers and CNNs (diffusion backbones) have increasing latency per step - even with KV-caching, the computational and memory complexity for a single step is $O(T)$, where T is the runtime. The complexity per step quickly becomes computationally infeasible with the growing context, making it necessary to use a cache of a few recent frames, preventing long-term dependencies. Conditioning the world model on the SSM state therefore preserves long-horizon information while keeping both memory and per-frame compute constant. In addition, we remind that while training, transformers still have a quadratic complexity, which is not the case with SSMs.

Baselines. We thank the reviewer for suggesting Diffusion Forcing. It is a strong transformer diffusion model that excels at long video generation. However, we find that it too has the same context limitation, innate to transformers. We used their official online demo and performed a long context test on all 16 of the images they made available to try (Demo 3 - Image -> Extremely Long Video). In our setup we do a 60 degree turn in place to the right, followed by a 60 degree turn to the left. This resulted in 10 generated frames. In all of our 16 tries we observed the content from the start that was hidden by panning the camera to be forgotten - in its place completely new content is hallucinated at the end of the generated video. We are not allowed to upload images or add links, but we encourage the reader to attempt this quick test.

Diffusion Forcing is targeted on long output generation, while we target consistency with long input (context) - even if generating a short future sequence. Our experiments with DIAMOND give us conclusions about the long-context abilities of our model. However, we recognize that, in future work, combining both paradigms can create a more capable model - both handling long generation and long context.

CSGO Qualitative Results. (1) As part of StateSpaceDiffuser, we train a light diffusion world model, under a fixed compute budget. As a result, frames in the long rollouts can occasionally appear with lower quality. Our contribution - the long-context mechanism via the SSM state, is decoder-agnostic: if a newer/larger conditional diffusion model is substituted, or if training time and sampling budget are increased, visual sharpness can improve without changing our method. Orthogonally, scaling the SSM (state dimension/heads and number of parameters) reduces the decay of high-frequency details carried over many steps, which also aids perceived sharpness. (2) Our primary objective is content consistency at long horizons. Therefore, the qualitative panels on Fig.6 and Sup. Mat. are in the challenging setting of forward-backward evaluation, with ensured enough motion to hide and reveal content. In these cases, the baseline produces inconsistent results while our method recalls the previously seen content, with a visual quality trade-off on the output. We will include qualitative examples in the final version of Sup. Mat. to demonstrate that the visual quality is better in the general case - on random sequences, without mirroring the action sequence (forward-backward).

Context Length. In training, we always use training sequences of a fixed size. At a first glance, it may seem that this could lead to overfitting and the reviewer rightfully raises the question about generalizeability on different context lengths. StateSpaceDiffuser is capable of operating on much longer context lengths than it has been trained on, without any fine-tuning required. We have demonstrated this in Sup. Mat. Sect. 2.1 (Lines 50-56), where we show that a model, trained exclusively on context length 16, significantly outperforms the baseline on context length 50 - without any finetuning. To expand this discussion, as the reviewer asked, we take our model, trained on context length 50 and test it on much longer context sizes. To this end, we generate a new Minigrid test set with sequence length up to 150. Results are shown below:

It can be seen that StateSpaceDiffuser successfully generalizes to hundreds of frames in the context. We observe visually reasonable visual and context recall quality - examples will be included in Sup. Mat.

Writing feedback. We thank the reviewer for pointing out the double reference issue, which we will make sure to fix in the final manuscript.

Computational Complexity. Mamba is known to have a linear complexity proportional to the sequence length, while trainsformers - a quadratic complexity over the (token) sequence length. While Mamba maintains a state, transformers and CNNs don't. To process the full sequence it has to be stored in its entirety in memory and processed at once. Even with KV-caching, the resulting linear complexity is only available in inference, and is linear per call - it does not support a streaming over a constantly increasing sequence. This is the case for current world models, including DIAMOND - a diffusion-based world model.

Measurements of the computational cost of the SSM and the diffusion model in our particular case can be seen at Lines 25-29 in The Supplementary Materials. Our SSM takes only 0.6% of all inference computations of the full model, for sequence size 16, and 1.8% for sequence size 50. To account for this computational cost in our StateSpaceDiffuser scores, we normalize them by multiplying by $1-0.006$ for sequence size 16 and $1-0.018$ for sequence size 50. We report the normalized scores and compare with the baseline:

Even after normalizing, significant gains are observed.

Generalization Over Longer Context. We train on fixed-length sequences, which reasonably raises the reviewer's generalization concerns. StateSpaceDiffuser operates on much longer contexts without finetuning: in Sup.Mat. Sect. 2.1 (L50–56), a model trained only at length 16 already outperforms the baseline at 50. At the reviewer’s request, we further evaluate a model trained at 50 on lengths 100 and 150 using a new MiniGrid test set (up to 150). Results are below:

It can be seen that StateSpaceDiffuser successfully generalizes to 100 and 150 frames in the future, keeping a significant gain over the baselines. These results could be further improved by finetuning on longer sequences.

Evaluation Across Seeds. As the reviewer suggested, we perform evaluation multiple times under different seeds. We run 4 times with different seeds for MiniGrid with context size 16 (computationally cheaper) and obtain $41.00\pm0.008$ Avg. PSNR, $40.52\pm0.019$ Fin. PSNR, $0.98\pm0.0004$ SSIM. We also perform evaluation over 4 seeds of a larger scale more expensive evaluation - our new 100 context length experiment from above. This results in: $37.99\pm0.002$ Avg. PSNR, $35.88\pm0.029$ Fin. PSNR, $0.98\pm0.000004$ SSIM. In both cases, the obtained metrics are extremely stable and consistent in between seeds (low std).

Evaluation Over Varying Visual Complexity. In our main paper, we evaluate on 3 different environment setups with increasing level of complexity - the very constrained Simple MiniGrid (Line 245), free navigation in a maze (Line 257), 3D first-person environment (Line 274). We keep the model scale (hyperparameters) the same across these environments and evaluate the performance on each. To further compare the performance of StateSpaceDiffuser across environments with different visual complexities, we generate 2 more variants of our MiniGrid dataset based on visual complexity. We define complexity as number of markers and complexity of the maze walls (values in the range $[1,5]$). We generate a dataset low complexity (200 markers, difficulty 3), and one with high complexity (450 markers, difficulty 5). Our original MiniGrid dataset lies in between in terms of complexity (360 markers, difficulty 4). We take our StateSpaceDiffuser pretrained model on context length 50 (middle complexity) and evaluate it on the new datasets with varying difficulties (without any finetuning):

User Study. CSGO is characterized by actions with variable motion unfolding over multiple frames. Long rollouts accumulate small motion mismatches into camera-view drift, so later frames need not match ground-truth pixels. As shown in Sup.Mat. (L127–137, Fig. 8), content often remains similar while viewpoint/details differ, and PSNR/LPIPS under-report this similarity. This is a known issue - see High-Resolution Image Synthesis with Latent Diffusion Models (p. 8); Image Super-Resolution via Iterative Refinement (p. 2). From Latent Diffusion: "simple image regression model achieves the highest PSNR and SSIM scores; however these metrics do not align well with human perception and favor blurriness over imperfectly aligned high frequency details.". Following common practice (e.g. EmuVideo, Stable Diffusion, Latent Diffusion), we therefore include human preference evaluations. We thank the reviewer for pointing out the mistake in the question's response. The exact instructions, shown to our volunteers is available on Line 150-156 of Sup.Mat.

MiniGrid Exploration Policy. The policy for generating the rollouts in MiniGrid is deterministic. We choose a sequence of 40 random markers (colors) and find the shortest valid path between them in the maze. This results in a very long path to follow - the actions are deterministic. Once 50 actions are complete into this path, the policy stops following the rest of the path and retraces the steps back to the beginning. This constitutes one rollout in our dataset. We will clarify this in our methodology section.

Separate Action Embeddings. Our training consists of two separate stages: 1) training the Long-Context Branch; 2) training the Generative Branch, conditioned on the Long-Context Branch. Therefore, we keep separate action embeddings in each branch to be trained in each stage for the needs of the particular branch. In this way we are able to preserve the modularity - e.g. replacing the state space model without sabotaging any shared action embeddings. We keep the dimensions in the Generative Branch as chosen in the baseline (DIAMOND), for our Long-Context Branch we use more compact dimensionality (16) to account for the lower amount of parameters and computational budget for this branch.

Robustness to Noise. We perform the experiment of adding noise in the middle section of the rollouts, suggested by the reviewer. In specific, we consider context length 50 in MiniGrid and add Gaussian noise (std 2.5) to the 11 frames in the middle (so noise remains symmetric in the sequence). We consider two cases: 1) adding noise to the SSM input only; 2) adding noise both to the SSM input and the diffusion model input. Results are shown below at different steps of prediction after the middle frame:

We observe that for the specific frames with added noise, the performance decreases. We observe on those frames that content can disappear and long context is not correctly recalled. However, in both cases, within 4 steps after the noisy frames (after frame 34 - 5 noisy frames + window size 4) the memory and content recovers and is correctly predicted, with stable performance until the last frame. Still, the lower scores suggest some loss in performance. The fact that memory recovers after the noise suggests a certain level of robustness to noise.

Writing Suggestions. Citation issues will be fixed in the final manuscript.

We are glad our earlier responses were useful and appreciate the opportunity to clarify these remaining points.

1. The experiments and conclusions - both the memory robustness of our method and the decrease of the PSNR on the specific frames with added noise, will be reported in our final manuscript. The differences on the predicted frames with the noisy input are observable, not in a form of noise but in a form of missing visual elements (walls, markers). We believe the effect could be reduced by image augmentation strategies or engaging a more noise-robust pretrained tokenizer.

2. The baseline results are available in the first three rows of the table, provided for varying visual complexity in our previous response. The provided results and the extended analysis will be included in the final manuscript. Similarly to how we evaluate our method, we took the pretrained baseline DIAMOND on middle complexity, and we also evaluate it on the new low and high complexity datasets - the same that we evaluated our method (StateSpaceDiffuser) as well. To provide some extra analysis: StateSpaceDiffuser outperforms the baseline on each of the corresponding dataset for visual complexity. Between visual complexities, the baseline significantly drops in performance for higher complexity, and to a lesser amount to lower complexity - a similar pattern as with StateSpaceDiffuser. We will include those results in our final Sup.Mat. with a reference from the main text. If any additional analysis of our visual complexity experiment results is needed, please let us know.

3. In terms of results stability/variability, we have provided results for context size 100 in the "Evaluation across seeds" section of our response - the results are very stable as can be seen by the very low standard deviation across 4 different seeds. For completeness, we mention the results here: $37.99\pm0.002$ Avg. PSNR, $35.88\pm0.029$ Fin. PSNR, $0.98\pm0.000004$ SSIM. In terms of dataset diversity/variability, we generated our longer context test set (up to 150 context length) with middle complexity with the same setup and size as we have previously generated our original Minigrid dataset - with random wall layouts, marker positions and colors per rollout and the same navigation path policy.

Thank you again for the constructive suggestions. We hope we addressed your remaining concerns, and we would be happy to answer any remaining questions.

We thank the reviewer for recognizing the importance of our topic, experimental results and the potential for use in future models. We address the reviewer's comments below.

Long Sequence Models Ablation. As the reviewer suggested, we have trained versions of the State Space World Model, but by replacing Mamba with LSTM and GRU for a context size 50 on our Minigrid setup. We add a linear layer on the input and output (dim 256, with ReLU activation) of the models. We are unable to use a standard transformer for a long image sequence because of the rising computational cost with each step - $O(N^2)$ complexity of the model in the general case. Furthermore, after trained, the transformer cannot generalize across a longer sequence than trained with. Results are shown below:

Mamba outperforms the other sequence models, not having to perform BackPropagation Through Time (BPTT), among other advantages. In addition, as shown in Tab. 2 in the main paper, Mamba also outperforms S4 - another popular state space model. Compared to other methods, Mamba excels at modeling long-range temporal dependencies, while remaining not computationally demanding. This is confirmed in the original paper that proposed Mamba - "Mamba: Linear-Time Sequence Modeling with Selective State Spaces", where Mamba is compared with other state space models (Table 1) and transformer language models (Table 3).

End-to-end Training. We trained the SSM and the diffusion model end-to-end in two setups: (1) from scratch, and (2) with an SSM pretrained on sequence dynamics followed by joint training. However, the best results were achieved by first train the SSM to capture long-range structure and then freezing it while training the diffusion decoder.

Joint training is challenging, as the components optimize on different timescales. The SSM needs many iterations of sequence-level supervision to stabilize a long-horizon representation, whereas the diffusion decoder optimizes a local denoising objective and rapidly finds a short-horizon solution that ignores a still-noisy SSM state (conditioning collapse). When we unfreeze a pretrained SSM for joint finetuning, decoder gradients quickly drift the SSM away from the representation that carries long-range cues; during this adaptation, the decoder again defaults to visual inputs and weakens reliance on the state. Freezing the SSM preserved its long-horizon information and led the diffusion decoder to make consistent use of it, yielding the strongest qualitative and quantitative results in our compute budget.

We find that this approach of separately training the two components has the important benefits of drop-replacing one of them without finetuning the entire system. For example, it allows us to use the same diffusion decoder that we trained for context size 16 with a new state space model, that we trained for sequence size 50, and the new resulting StateSpaceDiffuser model works better for longer context, without requiring any diffusion training. (Note that even without training a new state space model, StateSpaceDiffuser can generalize to much longer context out of the box - check answer of Reviewer dCL4 for more details.)

Reconstruction Quality. As part of StateSpaceDiffuser we train a lightweight conditional diffusion decoder under a fixed compute budget (no large pretrained backbone), which can yield frames artifacts late in long rollouts. Our contribution - the long-context conditioning mechanism via the SSM state, is agnostic to the choice of diffusion model. Replacing the decoder with a newer/larger one or training longer would improve visual sharpness without changing our method. Additionally, scaling the SSM (state dimension/heads/parameters) is expected to reduce high-frequency decay over many steps. Importantly, our primary objective is content consistency at long horizons. The qualitative results shown come from a forward–backward experimental setup with sufficient motion to occlude and later reveal content. On these challenging cases the baseline (DIAMOND) often becomes inconsistent or predicts incorrect content, while our model recalls the previously seen content - at some quality cost. For completeness, in the final Sup.Mat. we will also include qualitative examples from randomly sampled test sequences without the forward–backward constraint to show that visual quality in the general case does not degrade.

Writing Suggestions. We thank the reviewer for the suggested work - we will add it to our related work.