TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer

Thanks sincerely for your attentive review and profound comments. Below, we address the key concerns, with all citations consistent with the original paper.

Q1: Video Results and Enhanced Evaluation

Thank you for the feedback—we fully understand the importance of video results for evaluating visual quality. Videos with corresponding prompts are provided in the Video_Comparison folder linked anonymously in the supplementary (Line 9). This folder includes seven representative scenarios across both synthetic and real-world settings, including a ground truth benchmark in the following description. As shown, our method significantly outperforms IC-Light and VidToMe in temporal consistency, avoids the blurriness and unnatural lighting seen in COSMOS-Transfer1, and surpasses Slicedit in instruction adherence. TC-Light consistently delivers high-quality, stable results across dynamic scenes. All videos will be made publicly available on our paper's project page.

Additionally, following suggestions from reviewers phEG and xtUc, we include results on the Virtual KITTI 2 benchmark—a rare ground-truth-based relighting dataset with consistent geometry under varying lighting. We selected five scenes and relit them to match illumination of morning, sunset, rain, overcast, and fog settings, using COSMOS prompt upsampler to extract lighting and weather descriptors from the ground truth video as target prompts. Each sequence averages 281 frames at a resolution of 1248×384.

With access to ground truth, we replace CLIP-T and user preference with SSIM and LPIPS [26]. Results are shown below, following the same symbol definitions as Table 2 in the main paper. Our method outperforms all baselines in perceptual and structural similarity, as well as motion consistency (Motion-S, Warp-SSIM), while remaining efficient.

Q2: Illustration for MemFlow

Sorry for the confusion. We will revise the original phrasing from "where the first element is the flow ID (grouping pixels with shared optical flow)" to "where the first element is the flow ID (pixels connected by the optical flow predicted by MemFlow share the same flow ID)" for improved clarity. As suggested, we will also add a brief explanation of MemFlow in the Preliminaries section to provide proper context.

Q3: Claim Regarding Embodied AI

Thank you for highlighting this concern. To clarify, our mention of embodied AI was not intended to imply direct improvements to agent training, but to highlight a broader application context where dynamic video relighting serves as a foundational tool. Our method addresses critical limitations of prior work—namely, temporal inconsistency and computational inefficiency—making it possible to be applied to use cases beyond traditional domains like filmmaking or AR. To reduce confusion and improve precision, we will revise the keyword from "embodied AI" to "world-to-world transfer", and adjust the conclusion to: "endowing it with value and potential for broader application areas such as sim2real and real-world video scaling in embodied AI training and validation pipelines."

Q4: Meaning of Identical κ

Sorry for the confusion. Lines 161–165 present the general definition of κ. A crucial detail in interpreting "identical κ" lies in that the RGB component can be quantized. Without quantization, “identical κ” would require both identical flow connections and exact color matches—too strict for cases like view-dependent effects. By applying 7-bit quantization, we allow pixels within a slight color variation (~2/255) to be considered in gathering, enabling more robust grouping and filtering out unreliable flow connections. In our final implementation, κ uses flow ID and 7-bit quantized RGB. To enhance clarity, we will explicitly add this explanation regarding the physical meaning of "identical κ" to Line 165.

Q5: Physical Plausibility

The physical plausibility of our method is inherited from IC-Light, which is pre-trained on high-quality Light Stage data and has learned a physically grounded relighting process. Our main contribution lies in improving temporal consistency without altering the illumination priors embedded in the base model.

As detailed in Section 3.2, we introduce a video model inflation mechanism based on token merging/unmerging and decayed multi-axis denoising to enable temporal feature-level information exchange. Since these components do not alter the prior knowledge encoded in the base model, the distribution of edited illumination aligns with that of IC-Light while enhancing consistency across frames.

In Section 3.3, we propose a two-stage post-optimization strategy. The first stage smooths global exposure transitions using an appearance embedding, following practices in physically-based rendering methods like NeRF-W [CVPR’21] and 3DGS [SIGGRAPH’23]. The second stage refines local fluctuations without altering the overall lighting. Thus, our final results maintain the physically plausible qualities of IC-Light, while significantly improving temporal coherence. As shown in Figure 3 of the main paper and the video results in Q1, our illumination remains qualitatively aligned with IC-Light and VidToMe, but with fewer artifacts and greater temporal stability.

Thanks to the strong priors of IC-Light, our method focuses on temporal coherence and computation efficiency. Compared to optimization-heavy approaches such as Nerfactor [a] and InvRender [b], our post-optimization stage takes only ~2 minutes, with the entire pipeline completing in ~10 minutes—substantially faster than training NeRF or 3DGS models, as discussed in Lines 88–93 of the main paper. As suggested, we will incorporate the above discussion into the revised manuscript and add citations for both Nerfactor and InvRender.

We do appreciate your willingness to recommend our paper for acceptance and insightful advice. Below, we address the key concerns, with all citations consistent with the original paper.

Q1: Video Results and Enhanced Evaluation

Thank you for the suggestion—we fully agree on the importance of video results for assessing temporal quality. Videos with corresponding prompts are provided in the Video_Comparison folder linked anonymously in the supplementary (Line 9). This folder includes seven representative scenarios across both synthetic and real-world settings, including a ground truth benchmark (described in the following paragraph). As shown, our method significantly outperforms IC-Light and VidToMe in temporal consistency, avoids the blurriness and unnatural lighting seen in COSMOS-Transfer1, and surpasses Slicedit in instruction adherence. TC-Light consistently delivers high-quality, stable results across dynamic scenes. All videos will be made publicly available on our paper's project page.

Additionally, following suggestions from reviewers phEG and xtUc, we include results on the Virtual KITTI 2 benchmark—a rare ground-truth-based relighting dataset with consistent geometry under varying lighting. We selected five scenes and relit them to match illumination of morning, sunset, rain, overcast, and fog settings, using COSMOS prompt upsampler to extract lighting and weather descriptors from the ground truth video as target prompts. Each sequence averages 281 frames at a resolution of 1248×384.

With access to ground truth, we replace CLIP-T and user preference with SSIM and LPIPS [26]. Results are shown below, following the same symbol definitions as Table 2 in the main paper. Our method outperforms all baselines in perceptual and structural similarity, as well as motion consistency (Motion-S, Warp-SSIM), while remaining efficient.

Q2: Robustness Analysis

The influence of different submodules mainly reflects on the fluctuation of metrics. First, TC-Light inherits IC-Light's inability to relight hard shadows, as discussed in Section 4.4 of the paper. As shown in the Virtual KITTI video example in Q1, the tree shadows in ours_full.gif align with 0_input.gif rather than 1_groundtruth.gif. This results in SSIM deviating from the average value 0.59 to 0.47. Second, the performance is relatively stable to the design of the UVT index κ. Comparing rows 6, 8, and 9 of Table 3 of the main paper, the additional depth and instance mask clues bring a slight performance gain and drop. Third, unreliable flow estimation in some textureless regions may introduce artifacts, as discussed in Section 4.4 of the paper. Under the Video_Comparison/bad_case folder of Q1, an example from the AgiBot subset illustrates how unreliable flow leads to instability along the robotic arm's edge, with CLIP-T falling from the average value 0.275 to 0.259. We will expand these discussions in the revised paper to provide a clearer assessment of robustness, including visualizations of failure cases.

Q3: $E_t$ Illustration

Sorry for the confusion. We explain $E_t$ in Lines 143-144 that "Inspired by [26], we model $E_t$ as a 3 × 4 affine transformation matrix, initialized to the identity and optimized via Adam [29]." We would highlight this text and give an example of $E_t$ for an easier understanding in the modification.

Q4: CLIP-T Performance

The drop in CLIP-T reflects a trade-off between perceptual alignment and temporal consistency. Our post-optimization smooths flickering textures and illumination, slightly simplifying details and causing minor shifts in CLIP features. While CLIP-T decreases by ~1%, Warp-SSIM improves by over 20%. For example, in the video results of the CARLA subset in Q1, VidToMe and TC-Light achieve CLIP-T scores of 0.2556 and 0.2540, respectively, while Warp-SSIM improves from 83.90 to 93.66. As evidence, ours_full.gif is significantly smoother and more consistent than vidtome.gif.

We really appreciate your affirmation of our work and constructive advice! Below, we address the key concerns, with all citations consistent with the original paper.

Q1: Side-by-side comparison of TC-Light Modules

As suggested, we provide a side-by-side comparison table for clarity. Note that 3DGS is used for reconstruction instead of video editing. We will include this table and summarize our module distinctions at the start of Sections 3.2 and 3.3 to highlight our contributions.

Q2: Results on Groundtruth Benchmark

We appreciate the suggestion and agree that ground-truth evaluation remains scarce but essential in generative video relighting. To this end, we evaluate on Virtual KITTI 2, which renders consistent geometry across varied lighting conditions under autonomous driving scenarios. We selected five scenes and relit them to match illumination of morning, sunset, rain, overcast, and fog settings, using COSMOS prompt upsampler to extract lighting and weather descriptors from the ground truth video as target prompts. Each sequence averages 281 frames at a resolution of 1248×384.

With access to ground truth, we replace CLIP-T and user preference with SSIM and LPIPS [26]. Results are shown above, following the same symbol definitions as Table 2 in the main paper. Our method outperforms all baselines in perceptual and structural similarity, as well as motion consistency (Motion-S, Warp-SSIM), while remaining efficient.

Q3: Robustness Analysis

First, we evaluate TC-Light’s robustness under fast motion using the NavSim subset (5 sequences and ≥ 584 frames), controlling the total frames to 145 while varying sampling intervals. As shown in the table above, when motion speed increases, CLIP-T and Motion Smoothness remain stable (<2% fluctuation), while Warp-SSIM declines by ~5% under very fast motion due to less reliable optical flow. Second, in the scand subfolder of Q1's video results, we demonstrate TC-Light’s handling of heavy occlusion. As shown in ours_full.gif, our model correctly preserves the appearance of the same identity, while there is a minimal drop (<1%) in Warp-SSIM and Motion Smoothness compared to the SCAND subset average. These discussions would be supplemented to our revised manuscripts.

Q4: Details about the benchmark

Thank you for the valuable suggestion. Key details are as follows:

• The user study protocol is described in Section D of the supplementary, while dataset statistics appear in Table 1 (main paper) and Table S2 (supplementary). Dataset content, including scenarios and camera setups, is outlined in Lines 195–196 (main paper) and Lines 23–30 (supplementary). We plan to add thumbnails of each subset for clearer visualization.
• For clip selection, sequences shorter than 300 frames are used fully; otherwise, 300 consecutive frames are randomly sampled. Statistics are provided in Table 1 of the main paper.
• Domain balance is maintained between synthetic and real environments (25 vs. 28 videos), also balanced within sub-domains: autonomous driving (12 synthetic, 10 real), robotic manipulation (8 synthetic, 12 real), indoor navigation (5 synthetic, 6 real). The aerial subset is excluded from balance due to limited long dynamic drone videos in the simulation environment and serves mainly for generalization validation.
• Test clip separation is strictly ensured. IC-Light, trained solely on images, has never seen these video datasets. Our temporal inflation is zero-shot, and the post-optimization process is input-agnostic. Text prompts are randomly selected from IC-Light prompts or COSMOS-generated ones, with unreasonable prompts resampled (e.g., "a modern interior room" for a driving scenario). This selection process is fully independent of model development.
• This benchmark is solely for subjective/relative evaluation. But we also supplemented results on the Virtual KITTI dataset, which contains ground-truth video (see Q2). The table in Q2 sufficiently validates the superiority of our method's performance.

As recommended, we will add a dedicated section in the paper to comprehensively introduce the benchmark and include these details.

We sincerely thank you for the very professional comments. We are inspired and hope our discussion brings more insights. Below, we address the key concerns, with all citations consistent with the original paper.

Q1: Results on Groundtruth Benchmark

We appreciate the suggestion and agree that ground-truth evaluation remains scarce but essential in generative video relighting. To this end, we evaluate on Virtual KITTI 2, which renders consistent geometry across varied lighting conditions under autonomous driving scenarios. We selected five scenes and relit them to match illumination of morning, sunset, rain, overcast, and fog settings, using COSMOS prompt upsampler to extract lighting and weather descriptors from the ground truth video as target prompts. Each sequence averages 281 frames at a resolution of 1248×384.

With access to ground truth, we replace CLIP-T and user preference with SSIM and LPIPS [26]. Results are shown above, following the same symbol definitions as Table 2 in the main paper. Our method outperforms all baselines in perceptual and structural similarity, as well as motion consistency (Motion-S, Warp-SSIM), while remaining efficient. Related video comparisons are provided in Q2.

Q2: Video Results

Thank you for the suggestion—we fully agree on the importance of video results for assessing temporal quality. Videos with corresponding prompts are provided in the Video_Comparison folder linked anonymously in the supplementary (Line 9). This folder includes seven representative scenarios across both synthetic and real-world settings, including the ground truth benchmark in Q1. As shown, our method significantly outperforms IC-Light and VidToMe in temporal consistency, avoids the blurriness and unnatural lighting seen in COSMOS-Transfer1, and surpasses Slicedit in instruction adherence. TC-Light consistently delivers high-quality, stable results across dynamic scenes. All videos will be made publicly available on our paper's project page.

Q3: Reproducibility of TC-Light

Yes, we are planning to release the code upon acceptance. In fact, our code is already in a state where it can be open-sourced at any time. We have also used ChatGPT to improve the quality and readability of the whole repository. We hope the code release can benefit the research community.

Q4: Alternative Design Choices

We evaluate different compositions of κ and the use of learned index hashing (via vector-quantize-pytorch library). "inst." and "vq" denote instance masks and vector quantization, respectively. Rows 1, 3, and 4 match rows 6, 8, and 9 of Table 3 in the main paper.

Color proves essential in κ: flow+depth leads to a notable drop in Motion-S and CLIP-T compared with flow+color+depth. This is because even when pixels are close in 3D or flow-connected, they may belong to different objects or textures—color helps disambiguate such cases and avoid incorrect grouping. And the performance is relative stable to depth and instance mask clues.

Learned index hashing breaks spatiotemporal correspondence: we found this process introduced significant overhead while degrading Warp-SSIM and CLIP-T. The learned codebook often failed to capture true spatiotemporal correspondence, grouping unrelated pixels and receiving conflicting supervision. Hence, we excluded it in our model development.

Q5: Robustness Analysis

Here, we provide quantitative results under different flow conditions. The used data subsets align with that of the ablation study in the paper, and row 1 here comes from row 8 of Table 3 in the main paper. The noise level indicates the standard deviation of Gaussian noise added to the flow. As can be observed, the Motion-S and CLIP-T are relatively stable to the flow noise (drop less than 1%). From low to high noise, the drop of Warp-SSIM ranges from 2%-5%. Qualitatively, the frames become noisy, and flickering becomes more salient. We have also tried flow denoising supervised by warping error, but it brings 6-10 GB additional VRAM cost, and the results show no clear benefits. We have also tried FlowFormer++ during model development. As validated in [14], FlowFormer++ underperforms MemFlow. And as shown in the Table, a weaker flow estimator would also deteriorate the Warp-SSIM. Therefore, it is excluded from our model design.

Q6: Extension to Spatially Varying Lighting

Yes, the top row of Figure S1 (supplementary) demonstrates our model’s ability to handle spatially varying lighting. In the middle three images, a white car drives from left to right. Initially, it is illuminated by orange street lamps, reflecting an orange hue. As it moves right, its rear remains orange-lit, while the front becomes blue due to a nearby advertising screen. Eventually, the car is fully bathed in blue light. This dynamic lighting response indicates our model can correctly handle spatially varying light.

Q7: Limitation Analysis

As discussed in Section 4.4 of the main paper, the limitations are fourfold. First, TC-Light inherits IC-Light's inability to relight hard shadows. As shown in the Virtual KITTI video example in Q2, the tree shadows in ours_full.gif align with 0_input.gif rather than 1_groundtruth.gif. This results in an SSIM deviate from the average value 0.59 to 0.47. Second, the model struggles when the input resolution is significantly lower than that used to train IC-Light, leading to more severe flickering. For instance, downscaling the NavSim subset from 960×536 to 480×264 causes Warp-SSIM to drop from the average value 90.46 to 88.36, although CLIP-T remains at 0.304. Third, flow estimation becomes unreliable in textureless regions, introducing artifacts. Under Video_Comparison/bad_case folder of Q2, the example from the AgiBot subset shows how unreliable flow leads to instability along the robotic arm's edge, with CLIP-T falling from the average value 0.275 to 0.259. Fourth, the temporal consistency loss tends to smooth the texture of flickering areas. In the Carla subset of Q2, ours_full.gif shows smoother road and sky textures than iclight.gif, trading off realism for consistency. We will expand these discussions in the revised paper to provide a clearer assessment of robustness, including visualizations of failure cases.