UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting

We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.

Comparison with classical approaches. We agree that classical physically based rendering (PBR) methods can achieve high accuracy when high-quality 3D geometry and materials are available. However, in the absence of carefully authored scene content, methods like screen-space ray tracing (SSRT) often struggle to produce artifact-free shadows and reflections, as also noted in DiffusionRenderer [39].

In this work, we focus on enabling high-quality relighting directly from video without requiring full scene reconstruction. We agree that integrating our generative approach with explicit 3D information—such as traced shadow buffers—is a promising direction for improving physical accuracy, and we will discuss this in the revision as future work.

Video diffusion model vs. Image diffusion model. Video diffusion models have better capacities to model dynamic scenes and ensure temporally consistent results. This is especially important for effects like reflections and shadows, which may otherwise flicker, pop in/out or change shape abruptly when occluders enter or leave the frame. Please refer to our qualitative comparisons in the supplementary video, where we also include results from image-based baselines for comparison.

We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.

Joint modeling ablation. Our joint modeling formulation encourages the model to learn internal representations of scene structure, which improves generalization to unseen domains and mitigates shadow baking artifacts. We include a quantitative ablation in Table 3, and qualitatively demonstrate in Fig. 5 and Supplementary Video (0:44–0:59) that joint modeling effectively decomposes shadows and generalizes to challenging in-the-wild scenes.

Comparison with DiffusionRenderer. We re-implemented DiffusionRenderer using the same Cosmos backbone and trained it on the same synthetic and auto-labeled real-world datasets, denoted as DiffusionRenderer (Cosmos). DiffusionRenderer decomposes relighting into two stages: an inverse rendering step that estimates scene G-buffers, followed by a forward rendering step that synthesizes relit images conditioned on these estimates. This design introduces fundamental limitations: it is sensitive to errors in the predicted G-buffers (e.g., roughness) and struggles to capture complex material properties not encoded by G-buffers, such as transparency. We present quantitative comparisons in Tab. 1 and qualitative results in Supp. Fig. 2, showing that our joint modeling yields higher quality and greater flexibility when learning from diverse supervision sources.

Comparison with two-stages model. We experimented with an “image+albedo+lighting-conditional relighting model”. In practice, low-capacity versions of this design often either bias toward “flat” outputs resembling the input albedo, or suffer from “shadow baking” where shadows in the input image are preserved in the output. These limitations motivated our design: we increase model capacity via token-wise concatenation and explicitly model the joint distribution of scene intrinsics and relit appearance. Our single-stage formulation also enables faster inference compared to two-stage pipelines.

Predicting more material properties. We agree that this is a promising direction, and predicting additional properties could further improve the model’s understanding of scene structure and materials. Currently, memory requirements scale with the number of properties, making this expensive for high-resolution videos. We will further discuss it in the revision and leave it as a direction for future work.

We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.

Joint modeling ablation. Our joint modeling formulation encourages the model to learn internal representations of scene structure, which improves generalization to unseen domains and mitigates shadow baking artifacts. When evaluated on the in-domain synthetic test set (Table 3) or the MIT benchmark (PSNR 18.33 vs. 18.09), the gain from joint modeling may appear modest. However, its benefits are more evident in real-world out-of-domain scenarios. As shown in Fig. 5 and Supplementary Video (0:44–0:59), our joint modeling version effectively decomposes shadows and generalizes to challenging in-the-wild scenes. We will clarify this in the paper and add additional qualitative examples in the supplement.

Treating relit RGB as another “buffer”. While relighting may appear structurally similar to intrinsic prediction, we empirically observed that simply adding relit RGB as another output buffer often leads to “shadow baking”, where shadows in the input are directly copied into the output. This motivates our design choice: we increase model capacity via token-wise concatenation and explicitly model the joint distribution of scene intrinsics and relit appearance. This encourages the model to disentangle shading effects and produce controllable, high-quality relighting.

Predicting more material properties. We agree that this is a promising direction, and predicting additional properties could further improve the model’s understanding of scene structure and materials. Currently, memory requirements scale with the number of properties, making this expensive for high-resolution videos. We will further discuss it in the revision and leave it as a direction for future work.

Ray maps encoding. Compared to downsampling, encoding the ray maps can preserve more details and align with other conditions in latent space during the denoising process.

Dataset usage. We agree that constructing a more specific dataset, including semi-transparent or reflective objects, can help our model be more robust in these scenarios. Meanwhile, our auto-labeled data also helps our model generalize to more data domains, such as skin and fur (Fig. 4), which are all very challenging domains.

Dataset ablation. We report an ablation on the auto-labeled dataset in Fig. 6 and Tab. 4. While we agree that a full dataset-level ablation would provide additional insight, it is currently prohibitive due to the high computational cost of training large-scale video diffusion models.

We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.

Probe robustness. Conditioning our model on light-probes assumes distant lighting, which is indeed a limitation. We note that our model often produces plausible results for indoor scenes, but do not expect it to scale to extreme cases such as local spotlights. Our training data includes HDR environment maps with randomized rotations and intensity, enabling the model to follow the control of environment map condition, and generalize well diverse probes. In practice, we observe stable performance under rotated or clamped probes. We can add results with environment maps with a single lit pixel/OLAT config in the revision.

Auto-label quality visualization. We re-implemented DiffusionRenderer [39] using the Cosmos backbone [46] to label the G-buffers, and achieved high-quality estimation on diverse scenes. We appreciate the suggestion and will include additional visualizations of the auto-labeled dataset in the revised supplementary material. While the labels are “pseudo-groundtruth”, Fig. 6 shows that training on this data improves relighting quality, demonstrating the value of large-scale real-world data.

Synthetic dataset visualization. Our synthetic dataset was rendered using around 2,000 HDR environment maps. We use a path-tracer to render video clips under randomized lighting rotations, intensities, and color temperatures. We will show some representative examples and include more details of our synthetic data generation in the revision.

Geometry handling. Our model implicitly handles occlusions and synthesizes cast shadows as demonstrated in several scenes (e.g., tree shadows) shown in Supplementary video 0:44~0:59, Fig. 5, and Supp. Fig. 3. However, performance may degrade with thin structures or fast-moving occluders. Incorporating 3D geometry-aware priors (e.g., traced shadows from estimated depth or meshes) is a promising direction for future work.

Efficiency. During inference, UniRelight uses more memory (37.9 GB @ 480×848 pixels x 25 frames) than Diffusionrenderer (32.0 GB @ 448×832 pixels x 25 frames). This is due to the joint modeling of albedo and relighting in a single denoising pass. Our inference is faster than DiffusionRenderer, as shown in Supp. Tab. 2, thanks to the single-pass architecture. We acknowledge that runtime and memory remain deployment challenges. As future work, we plan to adopt memory-efficient architectures such as Framepack [1] and explore model distillation. While 1080p and 4K are currently infeasible memory-wise using the Cosmos backbone on current GPUs, we leave acceleration and support for higher resolutions as future work.

[1] Zhang, Lvmin and Agrawala, Maneesh. “Packing Input Frame Contexts in Next-Frame Prediction Models for Video Generation” Arxiv 2025.