Temporal In‑Context Fine‑Tuning for Versatile Control of Video Diffusion Models

We sincerely thank the reviewer for the encouraging and thoughtful feedback. We appreciate your recognition of the clarity, extensibility, and effectiveness of our method. Below, we address your questions regarding positional encoding for longer sequences, the potential for multi-task joint fine-tuning, and the feasibility of fine-grained frame-level control under our in-context learning framework.

---

(1) Positional Encoding. After concatenation of conditional , buffer and target frames, the total number of frames likely exceeds the original video length supported by the pretrained model. If the video length during fine-tuning differs from that of the pretrained model, it is unclear whether this would cause issues. It would be helpful to clarify how the positional encoding is handled for the longer frames.

We thank the reviewer for this important technical question. This is generally not a concern because recent pretrained video models are trained on videos of diverse lengths and can generate longer sequences using efficient attention mechanisms and/or autoregressive approaches. Our method adds only a few buffer and condition frames, resulting in relatively small computational overhead.

Furthermore, pretrained video models leverage Rotary Positional Encoding (RoPE), which effectively generalizes to sequence lengths beyond those seen during training. Since RoPE relies on relative rather than absolute positional information, it handles extended contexts without significant performance degradation.

Importantly, within the scope of our experiments, the total concatenated sequence length (condition + buffer + target frames) remained well within the normal sequence lengths supported by the pretrained backbones, so we did not encounter any issues. Nevertheless, as long video generation remains a challenging problem with rapidly evolving research, we believe there are opportunities for further exploration in conditioning longer video generation.

---

(2) Scaling. In this paper, each application requires a set of training data for fine-tuning. Is it possible to jointly fine-tuning on multiple tasks and achieve zero-shot performance at inference time?

While not directly within the scope of our current research—which focuses on efficient fine-tuning with minimal data and computation—we believe this is definitely feasible. In the image domain, the Kontext [1] model demonstrates how in-context fine-tuning can build a general image editing model capable of zero-shot performance.

Our temporal in-context fine-tuning method works effectively because it leverages the pretrained distribution of video models well, suggesting it could quickly adapt to diverse samples when trained on large-scale diverse datasets. One key difference would be that while our current approach requires paired videos (source/condition and target), scaling to multi-task zero-shot performance would likely require triplets of source, target, and text instructions to incorporate diversity and enable controllable generation. This represents an exciting direction for future research.

[1] Batifol, Stephen, et al. "FLUX. 1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space." arXiv e-prints (2025): arXiv-2506.

---

(3) Precise Control. If fine-grained control over each frame (e.g., conditioning each frame on a depth map) is needed, is it feasible under the proposed in-context learning framework?

We appreciate the reviewer's question about the scope of controllability. The TIC-FT framework is conceptually flexible enough to support such scenarios and would fall under the video-to-video (V2V) category presented in our paper. For example, one could structure the input as temporally arranged sequences of [condition frames (depth maps), target frames] or [condition frames (depth maps), buffer frames, target frames].

However, we should clarify that TIC-FT's primary design goal is not for tasks requiring dense, pixel-aligned conditional inputs like per-frame depth control. For such applications, more direct approaches—such as channel-wise concatenation of conditions or ControlNet-style methods where pixel-aligned condition features are directly added to latent frames—would likely provide more straightforward learning pathways for the model.

---

Once again, we are grateful for your valuable feedback. Your questions have prompted us to clarify important aspects of our design and explore directions for future extension. We hope our detailed responses have addressed your concerns and look forward to further discussions.

We thank the reviewer for the thorough and encouraging evaluation of our work. We appreciate your recognition of our method’s originality, clarity, and practical significance, as well as your valuable suggestions regarding baselines, temporal metrics, and long-video applicability. We address the reviewer’s concerns point by point, specifically focusing on (1) baseline selection, (2) the absence of fine-grained motion metrics, and (3) the restriction related to input sequence length.

---

Q: Why omit video-specific control methods (e.g., ControlVideo, VidControl)? Does TIC-FT outperform them under identical compute?

We want to clarify that ControlNet-based approached mentioned in the paper, is the same method as ControlVideo, and our method does outperform the baseline as presented in the paper, under less compute. (The ControlNet we used in our experiments is indeed a video-centric version adapted for a Video DiT architecture, functionally identical to the ControlVideo setup you suggested.)

Current leading methods for conditional video generation primarily follow two main paradigms. The first is the ControlNet-based approach, where a parallel (and sometimes trainable) encoder branch is added to the main model to inject spatial or temporal conditions. The second is the concatenation-based approach, where conditioning information, such as a reference frame's latent representation, is directly concatenated with the input noise tensor during the diffusion process and often requires finetuning the whole model.

To ensure our comparison was representative of the current state-of-the-art, we selected baselines that reflect these two dominant strategies. ControlNet (ControlVideo) was chosen as a representative example of the first paradigm, while Fun-pose, a popular and efficient variant of the second category, was included as a strong representative.

We acknowledge that additional methods exist, but we did not identify any work that significantly deviates from these two paradigms or demonstrates substantially superior performance compared to these established approaches. So we believe this selection provides a fair and robust comparison against the primary methods currently employed in the field. However, we are open to discussion and would be happy to include additional baselines if you have specific suggestions!

(We were unable to find TADA model you mentioned despite an extensive search. It would be grateful if you could provide a reference, and we would be happy to perform a comparison.)

You can also check out GPU Memory Usage Comparison:

As the table demonstrates, our TIC-FT method operates at a significantly lower computational cost for both training and inference. Despite this high efficiency, TIC-FT achieves superior performance compared to these baselines, as detailed in Table 1 and Table 2 of our main paper.

---

Q: Buffer frames reduce distribution shift but may cause motion flicker. Did you quantify temporal instability (e.g., FLD score)?

We appreciate the reviewer's focus on detailed temporal evaluation. As a quantitative measure, our main paper's Table 1 and Table 2 already include several VBench metrics that evaluate temporal qualities, such as subject consistency, background consistency, and motion smoothness.

In addition to these metrics, we believe the qualitative results provide the most direct evidence of temporal stability. We kindly ask the reviewer to refer to our supplementary video examples. These videos demonstrate that our method produces temporally clean and smooth results, without introducing the flicker or instability the reviewer was concerned about.

To more directly address the concern about motion and stability, we have conducted further evaluations using the new Video-Bench benchmark[1] (accepted at CVPR 2025), which provides human-aligned scores for motion effects and temporal consistency.

The results on Video-Bench are presented below:

Table: Evaluation on Video-Bench

Results demonstrate the benefit of our design, and temporal consistency shows our method is robust from temporal artifacts.

[1] H. Han et al., “Video‑Bench: Human‑Aligned Video Generation Benchmark,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

---

Q: The 10-second limit stems from memory constraints. Could sequence chunking or gradient checkpointing mitigate this?

We believe the limitation regarding long video generation is not specific to our method, but rather a common challenge and ongoing research area for all video generation models. Recent works have focused on longer video generation by employing more efficient attention mechanisms or autoregressive diffusion approaches. Gradient chunking, as you mentioned, would indeed help since the primary bottleneck is memory usage during chunk generation. Improving conditioning for longer videos represents an interesting direction for future research.

---

Q: Synthetic datasets (GPT-4o/Sora) may bias results. Does performance hold on complex real-world motions?

We appreciate the reviewer's concern for evaluation on real-world data. To clarify, our evaluation was not limited to synthetic data. For the action transfer task, we curated videos directly from the SSv2 (Something-Something V2) dataset, which is composed entirely of real-world videos of humans performing various actions.

Our method demonstrated strong performance on this real-world dataset. The qualitative results for this task, showcasing TIC-FT's effectiveness on complex, real-world motion, can be found in Figure 10 in the Appendix. If you are aware of any other real-world experiments that would enhance our evaluation, we would welcome your suggestions!

---

Once again, we are grateful for your thoughtful and balanced feedback. Your comments have helped us further strengthen our work, and we remain open to additional experiments and further discussion.

We sincerely thank the reviewer for the thoughtful evaluation and for highlighting both the strengths and the potential areas for clarification in our work. We address each of the concerns in detail and provide clarifications regarding the inference procedure, buffer frame configuration, and the handling of multiple conditional frames below.

---

L168–169 describe the inference procedure as “iteratively identifying the frames currently at the maximal noise level t and applying the video diffusion sampler exclusively to those frames”. It seems like the buffer frames need to be denoised iteratively, possibly one by one, because they have different noise levels? This seems to increase the inference time complexity? What's the inference time complexity of TIC-FT, and how does it compare to baseline methods?

We would like to clarify that our TIC-FT denoising process remains fully parallel across all frames and does not introduce additional iterative denoising steps, thereby incurring no significant overhead. To provide a clearer picture of the denoising procedure, we offer the following step-by-step visualization.

• C: Condition Frame (a clean frame at t=0)
• B1, B2, B3: Buffer Frames with initial target noise levels (e.g., t=25, t=50, t=75)
• T: Target Frames to be generated (starting at max noise, e.g., t=100)

---

Denoising Process of TIC-FT (Ours)

This process follows the formal rules described in Algorithm 1 and Equation 7 of the paper.

Frame Structure: [ C | B1 | B2 | B3 | T | T | ... ]

t_global = 100 (Start)

• Frame Timesteps: [0, 25, 50, 75, 100, 100, ...]
• Frames Being Denoised: Target Frames (T)

t_global = 99

• Frame Timesteps: [0, 25, 50, 75, 99, 99, ...]
• Frames Being Denoised: Target Frames (T)

... (This continues until t_global = 76) ...

t_global = 75

• Frame Timesteps: [0, 25, 50, 75, 75, 75, ...]
• Frames Being Denoised: B3 + Target Frames (T)

t_global = 74

• Frame Timesteps: [0, 25, 50, 74, 74, 74, ...]
• Frames Being Denoised: B3 + Target Frames (T)

... (This continues until t_global = 51) ...

t_global = 50

• Frame Timesteps: [0, 25, 50, 50, 50, 50, ...]
• Frames Being Denoised: B2 + B3 + Target Frames (T)

t_global = 49

• Frame Timesteps: [0, 25, 49, 49, 49, 49, ...]
• Frames Being Denoised: B2 + B3 + Target Frames (T)

... (This continues until t_global = 26) ...

t_global = 25

• Frame Timesteps: [0, 25, 25, 25, 25, 25, ...]
• Frames Being Denoised: B1 + B2 + B3 + Target Frames (T)

t_global = 24

• Frame Timesteps: [0, 24, 24, 24, 24, 24, ...]
• Frames Being Denoised: B1 + B2 + B3 + Target Frames (T)

... (This process continues down to t_global = 1) ...

t_global = 0 (End)

• Frame Timesteps: [0, 0, 0, 0, 0, 0, ...]
• Result: Final denoised sequence.

---

As the visualizations illustrate, our method requires no additional denoising steps. The only extra computation comes from additional condition and buffer frames, which is negligible when generating longer videos. For further clarity, visual examples of denoising are provided in Appendix Figures 8–12.

---

It is unclear to me how the number of buffer frames is determined during training and inference.

The choice of using three (latent) buffer frames as the default in our main experiments was based on a careful balance between generation quality and computational overhead, informed by the ablation study presented in our appendix (Figure 7).

Our ablation study on buffer frame count revealed the following trade-offs:

Too few buffer frames (e.g., one): Results in noisier transitions due to abrupt changes between condition and target frames, causing artifacts and degraded quality in target frames.

Too many buffer frames: Unnecessarily increases sequence length, leading to higher computational and memory costs, without further gain in generation quality. Additionally, excessive temporal gaps between target frames may reduce condition adherence.

We also observed that increasing the number of buffer frames enhances video dynamics in I2V tasks. This occurs because additional buffer frames create a temporal gap between the static input and target videos, leading to disentanglement between the two.

Intuitively, as the model further disentangles the target frames from the condition frame, the influence of the condition on the generated target video tends to diminish:

In conclusion, more buffer frames lead to disentanglement between condition and target frames. In I2V generation, since condition frames are static, additional buffer frames increase the dynamics of the generated video, while at the cost of reduced condition reflection. We determined that 3 buffer frames provide the optimal balance between memory usage, condition reflection, and dynamic degree.

---

It is unclear whether, in cases with multiple conditional frames, does the model only use the first conditional frame to construct the buffer frames? if so, how does the model use multiple conditional frames when constructing the buffer frames?

When multiple conditions are used, the buffer frames are constructed using only the last conditional frame. This design ensures a smooth transition from the condition sequence to the target frames. This process is illustrated in Figure 12 of the appendix.

---

Once again, we appreciate the reviewer’s constructive feedback and critical observations. Your feedback will be used to help us refine our explanations. We look forward to further discussions.

We sincerely thank the reviewer for the thoughtful and detailed feedback. Your insightful comments have helped us identify key areas to clarify and improve. We have carefully addressed each concern below.

---

(-) Small-scale evaluation set. Each task is evaluated on only 20 condition-target pairs, which may not be very representative of showing the whole picture of the performance superiority. Are there larger scale benchmarks or more standard ones that can be used for evaluation?

We would like to first clarify that the 20 condition-target pairs were used for training to demonstrate our method's few-shot efficiency. The quantitative evaluation reported in our initial submission was conducted using standard and diverse benchmarks including VBench [1], GPT-4o, and Perceptual similarity scores, with 100 unseen video samples per task. We did mention it in the manuscript, but will improve the camera-ready version for greater clarity!

To further strengthen our evaluation and address the reviewer’s concern, we have benchmarked our method on Video-Bench [2], a newly proposed large-scale benchmark recently presented at CVPR 2025. Video-Bench offers human-aligned evaluation metrics powered by multimodal LLMs, which demonstrate stronger correlation with human judgment compared to previous benchmarks such as VBench.

The results on Video-Bench are presented below:

Table: Evaluation on Video-Bench

As the tables show, our TIC-FT achieves competitive or state-of-the-art performance across most metrics on this large-scale, human-aligned benchmark (notably with less compute and better condition reflectance).

We believe these additional results on a large-scale, standardized benchmark effectively demonstrate the generalizability and robust performance of our method.

[1] Huang et al., Vbench: Comprehensive benchmark suite for video generative models

[2] H. Han et al., “Video‑Bench: Human‑Aligned Video Generation Benchmark,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

---

(-) Metrics: One drawback I could imagine from the temporal concatenation is the strong reliance of every generation frame on the input conditions (or condition shortcuts), which may lead to less dynamics in the output videos. VBench provides metrics beyond just consistency (including dynamic degree), but they are not reported in the tables.

We thank the reviewer for raising this important point. We provide the VBench Dynamic Degree metrics in the tables below for both I2V and V2V tasks, and show that our method substantially improve dynamic degree compared to other methods. While pixel-aligned conditioning methods such as ControlNet are known to degrade dynamic degree, temporal concatenation method (Ours) is less susceptible to this issue. Additionally, our fine-tuning approach leverages the inherent data distribution more effectively than alternative methods, limiting such degradation.

Table: I2V Evaluation on VBench Dynamic Degree

Table: V2V Evaluation on VBench Dynamic Degree

We further report the Dynamic Degree Discrepancy, computed as the difference between the Dynamic Degree of the Training Data and that of each generated video. (The best model should have the smallest absolute value of the discrepancy). The results show our method closely matches dynamic degree of the training data.

Table: I2V Evaluation on VBench Dynamic Degree Discrepancy

Table: V2V Evaluation on VBench Dynamic Degree Discrepancy

(-) Ablation study on the number of buffer frames.

We have performed additional quantitative analysis to derive insights on how the number of condition and buffer frames impacts the final output.

1. Impact of the Number of Condition Frames

We found that increasing the number of condition frames in I2V tasks lead to a marginal but consistent improvement in reflecting the condition's visual details. As shown in the table below, most of the similarity metrics show a slight upward trend with more condition frames.

2. Impact of the Number of Buffer Frames

We observed that increasing the number of buffer frames enhances video dynamics in I2V tasks. This occurs because additional buffer frames create a temporal gap between the static input and target videos, leading to disentanglement between the two.

We also observed that, as the model further disentangles the target frames from the condition frame, the influence of the condition on the generated target video tends to diminish, which is intuitive.

In conclusion, increasing the number of condition frames results in stronger reflection of conditioning on the generated frames, though the improvement is marginal. Additionally, more buffer frames lead to disentanglement between condition and target frames. In I2V generation, since condition frames are static, additional buffer frames increase the dynamics of the generated video, while at the cost of reduced condition reflection.

Given that the influence of condition frame count is marginal, it is optimal to use the minimum number of condition frames, as additional frames require more memory and computation. We found that 1 condition frame is optimal for I2V tasks. For buffer frames, we determined that 3 buffer frames provide the optimal balance between memory usage, condition reflection, and dynamic degree.

---

(-) I feel that the claim of "requires no architectural changes" is a bit off, as TIC-FT indeed requires additional LoRA parameters.

We agree with the reviewer that our original claim of "no architectural changes" is imprecise, as TIC-FT does introduce new LoRA parameters. We thank the reviewer for this point and will revise our manuscript with a more accurate description.

Our original intention was to highlight that, while our method introduces new trainable LoRA parameters, it does not alter the model's high-level architecture or its computational graph. (LoRA modifies the behavior within existing operational blocks rather than adding new, parallel network branches.)

(As the reviewer suggested) This preservation of the high-level architecture is what makes our TIC-FT framework inherently modality-agnostic and general-purpose, as no specialized modules are needed to handle different model architectures.

---

(-) I am unsure if the setting falls within the definition of in-context learning.

To clarify, our paper does not claim to implement in-context learning (ICL) in the traditional sense as defined in the context of LLMs. Instead, we define our work as "in-context finetuning", a related but distinct paradigm adapted for diffusion models. It is an efficient fine-tuning technique inspired from ICL, but involves finetuning, that leverages spatial or—in our case—temporal concatenation to facilitate information sharing across a sequence. Our novelty lies in extending spatial In-Context finetuning used for image generation to video generation, with Temporal In-Context Fine-Tuning (TIC-FT), along with techniques like adding buffer frames.

---

We sincerely appreciate your time and constructive feedback. We believe the additional results, clarifications, and analyses presented above effectively address your concerns and look forward to further discussion.