LuxDiT: Lighting Estimation with Video Diffusion Transformer

We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.

W: Clarity of Implementation Details. We appreciate the reviewer's feedback regarding implementation details. Supplements B.1 and B.3 provide detailed designs of our model, including the HDR MLP and modifications to CogVideoX. We will incorporate additional implementation details in the revised main paper.

W&Q5: Clarification of Figure 2. We apologize for the inconsistent description in the draft regarding Figure 2. The concatenation shown in Figure 2 and the addition described on Lines 159-160 for directional latent conditioning are essentially equivalent. Specifically, in CogVideoX, before being processed by transformer blocks, the concatenated latents $[\mathbf{z}^{ldr}, \mathbf{z}^{ldr}, \mathbf{z}^{dir}]$ are processed by a projection layer: $\mathbf{h} = W \cdot [\mathbf{z}^{ldr}, \mathbf{z}^{ldr}, \mathbf{z}^{dir}]$. This equation can be re-formulated as $\mathbf{h} = W_{0:2C} \cdot [\mathbf{z}^{ldr}, \mathbf{z}^{ldr}] + W_{2C:3C} \cdot [\mathbf{z}^{dir}]$, which aligns with the description on Lines 159-160. To eliminate this confusion, we will correct this description in the revised paper.

Q1: Stage II input representation. In Stage II, the noisy latent $\mathbf{z}_t$ is identical to that in Stage I. Initially, $\mathbf{z}_T$ is set to Gaussian noise with dimensions $[l, h, w, 2c]$. The DiT model continues to denoise this concatenated noisy latent $\mathbf{z}_t$ throughout Stage II training. The main distinction is that the denoised log part $\mathbf{z}^{log}_T$ is excluded from the diffusion loss computation if the input videos lack log maps.

Q2: VAE usage with single images. When the single image is used as the input, CogVideoX’s 3D causal VAE will encode the single-frame image as special independent latent tokens (8x8x1 compression for the first frame). The original CogVideoX model is also jointly trained with single-frame image generation to support this.

Q3: Directional embedding effectiveness. To enhance our model's ability to reason about the 3D orientation of environment maps, we utilize an additional directional embedding. This embedding allows us to augment our model training by randomly rolling or warping the input directional map, enabling the model to generate a corresponding environment map. Without this directional embedding, the model would only be able to predict environment maps with a fixed orientation, which could negatively impact its angular continuity. We additionally fine-tuned the model without using directional embedding and evaluated it on a subset of Polyhaven envmaps; the angular errors with three-sphere evaluation are reported in the table below, which justifies the effectiveness of directional embedding.

Q4: Inference procedure. At inference time, when $\mathbf{z}^{ldr}_T$ and $\mathbf{z}^{log}_T$ are initialized from Gaussian noise, we still concatenate the directional embedding along with denoising latents. However, the noise is not added to the directional embedding.

We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.

W1&Q1: HDR fusion. We found that per-pixel HDR fusion is sufficient for generating quality fused HDR maps. Our method exhibits fewer inconsistency issues compared to prior work, DiffusionLight. DiffusionLight estimates environment maps with varied exposure values through separate diffusion passes. In contrast, our method utilizes a single pass to obtain environment maps with two distinct tone-mappings. These two tone-mapped environment maps are decoded from different latent channels of the same denoised DiT tokens, which also enhances their spatial alignment.

To justify our design choice, we compared various HDR fusion approaches, including MLP, CNN, and a rule-based method, to justify the use of a simple MLP. The rule-based method applies the inverted Reinhard map for lights with intensity below 8, a linear interpolation between Reinhard and log maps for intensities between 8 and 16, and exclusively the log map for intensities greater than 16. The table below shows the RMSE

All three approaches achieve a similar level of accuracy, with the MLP approach performing slightly better than the other two approaches. Compared to the rule-based approach, we believe the neural approach can better handle numerical inconsistency after image uint8 quantization, and the potential data range overflow (e.g., lights beyond the pre-defined maximum intensity 10000).

W2&Q2: Clarification of the concatenation method. Our final model employs a dual concatenation approach. Specifically, the latents for $E_{ldr}$ , $E_{log}$ , and $E_{dir}$ are concatenated channel-wise. On the other hand, the latent for the input image condition $\mathbf{I}$ is token-wise concatenated with envmap tokens, as illustrated in Fig. 2. Regarding the ablation results in Table 7, Sec. 5.5, the "channel concat." variant concatenates all noisy and conditional latents along the channel dimension. For this variant, the envmap $\mathbf{E}$ is resized to match the input image $\mathbf{I}$, and no token-wise concatenation is used. To prevent confusion and misunderstanding, we will further clarify this design and enhance the consistency of our descriptions in the revised paper.

W3&Q3: Quantitative temporal analysis. We provided the metrics for measuring temporal consistency in Table 3. We computed the standard deviation (std) of per-frame error metrics for each video clip (L268-L270). For example, “PAE Std” in Table 3 measures the jittering scale of the peak light position in a video sequence.

W4&Q4: LoRA Scale Ablation. As suggested by the reviewer, we made an ablation on the LoRA Scale with input synthetic objects. The angular errors with three-sphere evaluation are shown in the table below

Contrary to the ablation made on scene images (Polyhaven crops), as the LoRA scale becomes larger, the corresponding lighting estimation accuracy becomes lower. We will also include visual results in the revised paper.

We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.

W1: Technical contribution. We thank the reviewer for prompting a deeper discussion of our contributions.

LuxDiT tackles two persistent challenges in HDR lighting estimation: (1) predicting accurate intensity and direction from spatial inputs, which requires inferring global, non-spatial outputs (360° panoramas) from local, spatial visual cues; and (2) recovering angular high-frequency details consistent with the scene content.

Our solution combines: (1) a scalable generative architecture that leverages a powerful pre-trained video diffusion model and a carefully curated large-scale synthetic dataset; (2) dual-tonemapped latent representations for HDR structure; and (3) LoRA fine-tuning on real HDR panoramas to improve semantic alignment. While each component builds on existing techniques, their integration is purposefully designed for the unique demands of lighting estimation.

To our knowledge, LuxDiT is the first end-to-end training method to formulate lighting prediction as a conditional generative diffusion task. We conduct extensive evaluations and ablations. LuxDiT reduces lighting direction error by 45% on Laval Outdoor (Table 2) and improves temporal consistency on video input (Table 3), enabling reliable use in downstream tasks such as virtual object insertion. We will release our models, dataset, and code to support reproducibility and future work.

W1: Dual-Tonemapping Light Representation. The use of multiple tonemapped light maps for HDR representation is common in rendering tasks. Similar approaches have been applied in prior methods for both lighting estimation (e.g., DiffusionLight) and relighting tasks (e.g., Neural Gaffer).

An alternative choice for our HDR light representation is to follow DiffusionLight’s multi-pass, multi-EV tonemapping. However, this design choice also falls short of our requirement for higher dynamic range environment maps with spatially coherent content. DiffusionLight's reliance on separate diffusion passes for estimating environment maps with varied exposure values introduces spatial inconsistencies. Its multi-EV HDR fusion results in a limited dynamic range, with a maximum intensity of 32. In contrast, our proposed method, LuxDiT, addresses these limitations by jointly denoising dual-tonemapped environment maps within the same latent tokens. This approach significantly enhances spatial consistency during HDR fusion. Furthermore, LuxDiT utilizes Reinhard and logarithm tonemapping functions, which provide sufficient sampling to cover a broad range of lighting scenarios, from rich LDR content to high-contrast spotlights (as shown in Supplementary Fig. S1). This enables HDR reconstruction with a peak intensity of up to 10,000.

W2: Significance of quantitative improvements. We evaluated our method on various benchmark datasets with different evaluation metrics. Specifically, on Laval Outdoor scenes, our model substantially enhances peak light direction estimation quality, reducing the error by 45% (Table 2). Furthermore, it improves video lighting consistency by decreasing temporal jittering by up to 81% (Table 3, "std" terms).

The reviewer's concern regarding metric inconsistency primarily stems from the Laval Indoor experiment. As stated in the paper (L242-L245), our model was not trained on the Laval Indoor dataset. This dataset exhibits a notable shift in color and intensity distribution compared to our training set. In contrast, the baseline method, StyleLight, is exclusively trained on Laval Indoor. While DiffusionLight does not directly use Laval Indoor, the Text2Light model, which generates its training data, is trained with Laval Indoor data. Despite these factors, our method demonstrates strong robustness by still achieving competitive metrics on the Laval Indoor dataset.

Regarding statistical noise and confidence interval, as also mentioned by R1, we evaluated LuxDiT three times using different random seeds and reported the average metrics. For brevity, we omitted the standard deviations in the main paper. Below, we provide the standard deviations for the metrics reported in Table 1 of the main paper. As shown in the table, the variance of metrics is small enough, not affecting the comparison results shown in the paper.

W3&Q1: Evaluations on other key baselines. We thank the reviewer for highlighting broader categories of lighting-related methods. Our work focuses specifically on single-view, geometry-free estimation of scene-scale 360° HDR environment maps.

Methods like NeRFactor operate under different assumptions. They typically require multi-view posed images and geometry to optimize lighting, shape, and material, and are designed for object-centric, per-scene optimization. Our method, conversely, focuses on generalizable, image-conditioned lighting prediction, making direct comparisons on standard lighting estimation benchmarks (e.g., Laval, PolyHaven) infeasible due to the distinct input conditions and scale of light estimation (object vs. scene).

We thank the reviewer for mentioning inverse rendering models for the lighting estimation. We did not deliberately exclude the inverse rendering models in our evaluation. The input conditions for inverse rendering models and image-based lighting estimation are very different. Inverse rendering requires additional multi-view posed images and 3D geometry for estimating light probes, and such light estimation is usually on object scale instead of scene scale. It is very difficult for us to report inverse rendering models’ metrics in Table 1.

That said, we include preliminary comparisons with inverse rendering methods (e.g., NVDIFFREC and NVDIFFRECMC) in Supplementary Sec. C.2 (Fig. S5). We also evaluated our method on the Objects-with-Lighting dataset [1] against NeuS+Mitsuba, as reported in the table below.

Our method is not expected to completely surpass inverse rendering approaches. Instead, we believe that integrating our generative model with inverse rendering methods could further improve lighting estimation.

The reviewer mentioned InverseRenderNet, an inverse rendering model that estimates lighting using low-frequency spherical harmonics (SH) from a single image. Our paper compares against models that also predict low-frequency lighting from RGB input (e.g., H-G et al. in Tables 1 & 2), and LuxDiT surpasses their performance. EMLight [2] is another method that relies on low-frequency light as guidance; the StyleLight paper included EMLight in its comparison, and EMLight did not outperform StyleLight. A comprehensive evaluation of earlier methods like InverseRenderNet is beyond the scope and time constraints of this rebuttal, but we will incorporate a discussion of its relevance to our work in the revised version.

Q2: Camera parameters for LDR crops. We uniformly sample the FOV of the perspective camera within the range between 45 and 80 degrees. The camera elevation angle is sampled between -10 and +10 degrees.

Q3&Q4: Robustness and generalization to real-world scenes. In the paper, we used the virtual object insertion rendering to show our model’s robustness and generalization to real-world scenes. We show the metrics in Table 8, visual rendering results in Fig. 1, Fig. 4, and Supplementary Fig. S11, S12. Note that all the Waymo driving scenes can be treated as in-the-wild real-world outdoor captures (not cropped from any panorama). Our model effectively estimates high-contrast sunlight, allowing for the accurate casting of sharp shadows from inserted objects.

[1] Ummenhofer, Benjamin, et al. "Objects with lighting: A real-world dataset for evaluating reconstruction and rendering for object relighting." 3DV, 2024.

[2] Zhan, Fangneng, et al. "Emlight: Lighting estimation via spherical distribution approximation." AAAI, 2021.

I agree with Reviewer GYdA30 that dual-tonemapping is not a novel contribution of this work.

However, I believe the proposed method for conditioning a diffusion model to generate artifact-free environment maps—particularly avoiding the common center artifacts seen in DiffusionLight—is a valuable contribution. This improvement is evident in Figure 3 of the paper and in comparison to qualitative results from the DiffusionLight paper.

Thanks for the rebuttal. My concerns have been addressed. But the novelty is still not convincing to me. I will raise the points up to 3.

We thank the reviewer for their valuable feedback, and appreciate that they recognize our contributions. We respond to the comments and questions below.

W1: Contribution. We thank the reviewer for prompting a deeper discussion of our contributions.

LuxDiT tackles two persistent challenges in HDR lighting estimation: (1) predicting accurate intensity and direction from spatial inputs, which requires inferring global, non-spatial outputs (360° panoramas) from local, spatial visual cues; and (2) recovering angular high-frequency details consistent with the scene content.

Our solution combines: (1) a scalable generative architecture that leverages a powerful pre-trained video diffusion model and a carefully curated large-scale synthetic dataset; (2) dual-tonemapped latent representations for HDR structure; and (3) LoRA fine-tuning on real HDR panoramas to improve semantic alignment. While each component builds on existing techniques, their integration is purposefully designed for the unique demands of lighting estimation.

To our knowledge, LuxDiT is the first end-to-end training method to formulate lighting prediction as a conditional generative diffusion task. We conduct extensive evaluations and ablations. LuxDiT reduces lighting direction error by 45% on Laval Outdoor (Table 2) and improves temporal consistency on video input (Table 3), enabling reliable use in downstream tasks such as virtual object insertion. We will release our models, dataset, and code to support reproducibility and future work.

W1&Q4: Rendering quality evaluation with estimated lighting. We use the lighting estimated by LuxDiT to render virtual objects for insertion into real scenes, as described in the main paper Sec. 5.4. Qualitative results are shown in Fig. 1, Fig. 4, and Supplementary Figs. S11 and S12. In response to reviewer W1, a user study was also conducted to evaluate perceptual quality (further details in Supp. Sec. B.4). As shown in Table 8, renderings generated with LuxDiT's lighting were favored over those produced by baseline methods.

W2&Q3: Variance of metrics. We thank the reviewer for pointing out the lack of variance reporting. Due to the stochastic nature of diffusion models, we evaluated LuxDiT three times using different random seeds and reported the average metrics. For brevity, we omitted the standard deviations in the main paper. Below, we provide the standard deviations for the metrics reported in Table 1 of the main paper.

As shown in the table above, the variance of metrics is small enough, not affecting the comparison results shown in the paper.

W3&Q2: Additional evaluation on TCC or CUBE++. We thank the reviewer for suggesting additional datasets for evaluation. We performed an additional experiment using 100 images from Cube++. We applied both DiffusionLight and our method to estimate lighting from each image, then rendered the left and right white faces of the SpyderCube under the estimated illumination, assuming purely diffuse reflectance. We compared the rendered face colors to the provided ground truth and reported RMSE and angular error. Our method outperforms DiffusionLight across all reported metrics.

We note that TCC and Cube++ are primarily designed for color constancy rather than 360° HDR lighting estimation, and the reflective spheres in these datasets lack accurate HDR ground truth or consistent annotations. As such, this experiment is inherently approximate and should be considered a supplementary evaluation.

Q1: More details on our dataset. We thank the reviewer for their interest in our provided dataset. Details of the dataset curation process, including selection criteria, rendering setup, and preprocessing steps, are provided in Supplementary Sec. B.2. This dataset will also be released.

We appreciate the reviewer’s pointer to Cube++ annotations and the additional resources. In our rebuttal-phase Cube++ experiment, the unexpectedly high angular errors for both LuxDiT and DiffusionLight were caused by a mismatch between the ground truth face colors we used and the JPEG images we tested on.

Specifically, we mistakenly sampled ground truth values from Cube++/auxiliary/extra/gt_json/*, which are derived from RAW images, whereas our methods (and baseline DiffusionLight) operate on processed JPEG images with different color responses.

For example, in Cube++ image 00_0001, the ground truth left-face color from the RAW-based JSON is (0.18, 0.46, 0.35), while the corresponding JPEG face color is approximately (0.32, 0.33, 0.35). LuxDiT’s rendering for that face is (0.30, 0.33, 0.37), yielding an angular error of 17.10° using RAW-based colors, but only 2.80° when compared correctly to JPEG-based colors.

We have re-evaluated using the correct color sampled from JPEG image with face annotations from Cube++/auxiliary/source/JPG.JSON/*, without re-running the model. The corrected metrics are shown below. Both methods now achieve much lower angular errors, with LuxDiT clearly outperforming DiffusionLight, achieving <5° on both faces.

We thank the reviewer for bringing this to our attention. We will include the corrected evaluation in the revised supplementary material and discuss it in the main paper.