IntrinsiX: High-Quality PBR Generation using Image Priors

Motivation for Image-Space PBR Priors

Text-based generative models can act as a generic prior over images, which has shown great value for numerous downstream tasks: improved intrinsic image decomposition(IID $29$ , RGBX $67$ ), depth estimation (Marigold $Ke et al CVPR 2024$ ), etc. Our goal is to establish a similar generic prior for PBR materials to support downstream applications like UV-based texturing.

The key challenge of the field of PBR generation is the lack of large-scale, diverse datasets of PBR UV textures. Unlike RGB data, material datasets are scarce, and crucially, consistent UV parametrization (mapping from object coordinates to texture coordinates) across assets practically does not exist. This makes training direct generative models in UV texture space infeasible.

Instead, we learn a prior in image space. To distill image-space information into texture-space, we use SDS, which aggregates image-space signals from multiple views. This enables PBR UV texture generation for 3D assets, even though trained with relatively small amounts of image-space PBR data. In contrast, decomposition methods require an image as input and it is not obvious how one would synthesize consistent PBR UV textures for novel 3D assets.

Inherent Ambiguity

We believe that intrinsic image decomposition methods are constrained to match the appearance of the input image. Thus, the model can learn to rely more on the input, instead of focusing on staying in the PBR distribution; therefore, facing an out-of-distribution input image poses a challenge to these models to produce faithful PBR maps, corresponding to the respective distributions.

L90 and L298 are also grounded on our empirical observations of our experiments. Concretely, the decomposition baselines perform well on in-domain samples (indoor scenes), but struggle to generalize out-of-distribution. In contrast our method does not have such limitations (see Tab. 1, Fig.6,7). Thus, we argue that decomposition is inherently more ambiguous than our task of directly generating the PBR maps.

ID vs OOD

Our diffusion-based baselines are only trained on indoor rooms, partially or fully on the InteriorVerse dataset. These methods perform well on indoor scenes, since this is in-domain for them. Our goal is to generate unconstrained scenes in diverse domains (e.g. single objects, composited scenes, outdoor, nature, etc.); therefore, we curated a dataset of complex scenes from various domains (OOD dataset). Since the baselines are image-conditional, when facing an out-of-domain input, their performance decreases. In contrast, our method does not depend on an input image, but rather learns the distribution of the PBR properties while maintaining the prior knowledge of the base model; thus, it can better generalize, as our qualitative and also quantitative results show.

Additional Results

We extend our test set and re-evaluate all the baselines and our method. For in-domain experiments, we use the full InteriorVerse test set (2595 samples, as done in IID $29$ ). Evaluating the out-of-domain performance is challenging since there exists no large-scale PBR dataset of complex scenes from various domains. To evaluate one specific domain, we use the single-object G-Buffer ObjaVerse $Zuo et. al. ECCV 2024$ dataset. We take 1000 samples from the diverse ``Daily-Used" category. To show that these numbers are representative, we evaluate the FID with a different number of samples. The relative improvement over baselines align with the original FID numbers in the main paper and the metrics are consistent. Our method achieves better generalization to out-of-distribution domains.

We provide three additional experiments to better support our claims (as also requested by the reviewers).

• Ours w/ BRDF-sampling: we sample more diverse lighting directions using BRDF-sampling during our rendering loss to provide more gradient signal about the BRDF behavior. This variant slightly improves our generalization results, showing the importance of lighting direction sampling.
• IID+FLUX LoRA: we re-train the IID baseline with FLUX as backbone and use LoRA layers similar to our method. This improves the fidelity, but this still does not reach the quality of our method, showing that learning a generalizable prior over PBR distributions is beneficial for generalization. For more details, we refer to our rebuttal to reviewer 48oZ.

We are happy to include these detailed results.

Image-conditional IntrinsiX

Our model can be made image-conditional in a similar fashion as for our downstream task of normal-conditioned PBR generation (Fig. 5, 15): we add the RGB image as input and apply input dropout during fine-tuning (as in L228). Our image-conditional method produces sharper results and outperforms decomposition baselines on OOD data (see Ours w/ Image-cond below). This demonstrates the benefits of learning a generative prior over PBR modalities.

# Samples vs FID on InteriorVerse

# Samples vs FID on G-Buffer ObjaVerse

FID Metrics

Writing

We fixed the typos as suggested and clarify eq. (3): \\hat{\\mathbf{I}} \= f(\\omega\_o; \\omega\_i; \\hat{\\mathbf{X}}\_0) \\cdot L\_i \\cdot(\\hat{x}\_n^T \\omega\_i)

Related works

The work of Xue et al. (Diffusion-based G-buffer Generation and Rendering, arXiv 2025) addresses text-conditional complex PBR generation. They train a shared ControlNet for Stable Diffusion v2 and use a diffusion renderer for editability. In contrast, our method first learns a prior over the PBR properties independently, then aligns them with cross-intrinsic attention using a fixed renderer to ensure compatibility with standard rendering engines. We will include a thorough discussion; however, at the same time we want to highlight that this work is concurrent (almost appeared at the same time on arXiv as our submission).

Single-material generation and capture

Single-material generation methods focus on texture generation; they cannot handle complex environments. ``SVBRDF capture” uses a feed-forward model with a similar rendering loss using the reflected viewing direction of a random surface point for lighting. Orthogonal to this direction, we aim to generate the PBR properties of complex scenes in image space to enable downstream applications. We introduce a rendering loss to the diffusion framework using a roughness-weighted importance sampling for the lighting direction.

Cross-frame attention

We discuss related approaches that use cross-frame attention in L77-80. Tune-A-Video similarly uses cross-frame attention, but as other multi-view image generation methods, they have the same modality across the ``frames" (=batch elements); thus, they can share the LoRA weights. MTFormer applies cross-task attention on the dense prediction heads of a deterministic transformer model. In contrast, our method uses cross-intrinsic attention, where all the LoRAs are different, but still tuned together for a generative task.

We will include these discussions.

Comparison against Decomposition

To the best of our knowledge, there is no other text-conditional PBR map generation method that operates on complex, unconstrained scenes in image space; therefore, we compare to decomposition methods. We agree that PBR generation is less constrained than decomposition, and our results demonstrate this: direct PBR generation is better suited for high-quality PBR generation (Tab.1, Fig.6,7).

Additional Results

We extend our test set and re-evaluate all the baselines and our method. For in-domain experiments, we use the full InteriorVerse test set (2595 samples, as done in IID $29$ ). Evaluating the out-of-domain performance is challenging since there exists no large-scale PBR dataset of complex scenes from various domains. To evaluate one specific domain, we use the single-object G-Buffer ObjaVerse $Zuo et. al. ECCV 2024$ dataset. We take 1000 samples from the diverse ``Daily-Used" category. To show that these numbers are representative, we evaluate the FID with a different number of samples. The relative improvement over baselines align with the original FID numbers in the main paper and the metrics are consistent. Our method achieves better generalization to out-of-distribution domains. We are happy to include these detailed results.

# Samples vs FID on InteriorVerse

# Samples vs FID on G-Buffer ObjaVerse

FID Metrics

Image-conditional IntrinsiX

Our model can be made image-conditional in a similar fashion as for our downstream task of normal-conditioned PBR generation (Fig. 5, 15): we add the RGB image as input and apply input dropout during fine-tuning (as in L228). Our image-conditional method produces sharper results and outperforms decomposition baselines on OOD data (see Ours w/ Image-cond above). This demonstrates the benefits of learning a generative prior over PBR modalities.

Lighting Direction Sampling

We sample light directions that maximize specular highlights to provide strong gradients about reflectance. We train a BRDF-sampled variant (Disney model) for more diverse lighting directions (denoted as Ours w/ BRDF-sampling above*)*. This yields slightly worse results on the in-domain dataset, but improves generalization, showing that lighting direction sampling is crucial for our task.

Model Inconsistency

We re-trained the IID $29$ baseline using the FLUX backbone, both with full fine-tuning (as in IID) and with LoRA (matching our model, 300M trainable parameter). Stronger backbone improves results. Full fine-tuning performs better in-domain (InteriorVerse) but generalizes worse (IID+FLUX Full). LoRA helps toretain the original prior and improves generalization (IID+FLUX LoRA). While sample sharpness improves, IID still shows artifacts (e.g., washed or misaligned textures). We believe image-conditioned models struggle with OOD generalization due to over-reliance on the input image, whereas our method learns a prior over PBR distributions that generalizes better. We are happy to include an extensive version of these experiments.

IID + FLUX

Rough and Metal Results

We agree that evaluating material properties alone is challenging, which is why we include renderings and relighting videos and also a user study assessing specular and rendering quality (Tab. 1). The first four rows of Fig. 12 show our first-stage model, trained without rendering loss or cross-intrinsic attention, hence the predictions are incoherent and do not understand the rough and metal properties well. After stage two (bottom four rows), we can see clear improvements. The horse's tail and the guitar soundhole are not metallic anymore (guitar strings correctly remain metallic). The helmet visor, guitar player's eye and the windows of the cabin should have glass-like reflections, which can be best matched with metallic (mirror-like) reflection. Training with more diverse and high-quality PBR maps should improve the generalization.

User Study

Since PBR quality is difficult to assess, user studies naturally exhibit a large uncertainty. However, we rate each method separately, and uncertainty of the different methods are close to each other. Thus comparing the mean values is fair.

Details on LoRA Finetuning

Each LoRA has ~75M params (0.625% of the base model), giving a total of 225M params. It is not necessary to fine-tune all layers, we insert LoRAs only into attention blocks, augmenting the to_q, to_k, to_v, and to_out layers. While we explored skipping some blocks to reduce parameters, we found that cross-intrinsic attention performs best when LoRA is applied throughout.

Dataset Size

LoRA finetuning can adapt T2I models effectively with few high-quality samples (see LoRA paper Sec. F.3). However, available larger PBR datasets are domain-specific (indoor). While images might become more diverse, the prompts do not vary that much, so the LoRA links style mainly to that domain. Therefore, out-of-distribution prompts (Fig. 9) don’t adopt the new style and remain shaded RGB images. The ideal solution is a large PBR dataset, with diverse, complex domains, which currently does not exist. Adding a new domain to the training improves the generalization, as discussed above (Rough and Metal Results).

Rendering

During training, we process only a single diffusion timestep, avoiding backpropagation through all steps, which enables training with a rendering loss. Starting from a noisy sample, a single-step (x0) prediction produces clean PBR maps that receive useful gradients when rendered. At inference, multiple lights can be rendered using a fixed forward renderer, as demonstrated in our scene texturing results with complex lighting (supplementary video at 1:25).

SDS

Flat colors often occur in SDS methods due to averaging predictions. To avoid this, high guidance scales are used but cause oversaturation (ProlificDreamer $Wang et. al.$ Fig.3). This is unacceptable for rough/metal as it alters reflectance. Thus, we use lower guidance and apply an observation-frequency weighted loss and normalized flow direction to preserve details (L983-990). Other works that overcome this limitation can be additionally applied to our downstream application (e.g., LoRA adaption in ProlificDreamer, consistent noising in SyncDreamer $Liu et. al.$ ).

Limitations

We’ll expand Sec. C and add a figure showing failure cases (flat decompositions, ambiguous metallic, noisy normals). T2I models have shown value as generic image priors for numerous tasks, e.g. improvements in intrinsic image decomposition. Our goal is to get a similar prior for PBR maps, i.e. focusing on broad domain coverage, and not on exact prompt precision.

Writing

We adopt the suggestions to improve the clarity of our exposition/method details and fixed the typos in L91,97,201,124,129,201.

Formatting

We will fix the formatting issues. Since other NeurIPS papers (e.g., “Light Field Networks” $Sitzmann et al., NeurIPS 2021$ ) use half-column figures, we believe this is allowed.

We thank again for the reviewer's feedback and hope our rebuttal addressed their questions. Please let us know if there are still open concerns.

Dear Reviewer,

Please respond to the rebuttal from the authors. Please note that, this year it is not allowed to finalize without discussion with authors.

Best,

Your AC

FID Metrics

We extend our test set and re-evaluate all the baselines and our method. For in-domain experiments, we use the full InteriorVerse test set (2595 samples, as done in IID $29$ ). Evaluating the out-of-domain performance is challenging since there exists no large-scale PBR dataset of complex scenes from various domains. To evaluate one specific domain, we use the single-object G-Buffer ObjaVerse $Zuo et. al. ECCV 2024$ dataset. We take 1000 samples from the diverse ``Daily-Used" category. To show that these numbers are representative, we evaluate the FID with a different number of samples. The relative improvement over baselines align with the original FID numbers in the main paper and the metrics are consistent. Our method achieves better generalization to out-of-distribution domains.

We provide three additional experiments to better support our claims (as also requested by the reviewers).

• Ours w/ Image-cond: we make our model image-conditional similarly, as we did for our normal-conditional model during the SDS-based texturing (L228). This variant maintains sharp results and outperforms baselines on OOD data, demonstrating the benefit of a learned PBR prior.
• Ours w/ BRDF-sampling: we sample more diverse lighting directions using BRDF-sampling during our rendering loss to provide more gradient signal about the BRDF behavior. This variant slightly improves our generalization results, showing the importance of lighting direction sampling.
• IID+FLUX LoRA: we re-train the IID baseline with FLUX as backbone and use LoRA layers similar to our method. This improves the fidelity, but this still does not reach the quality of our method, showing that learning a generalizable prior over PBR distributions is beneficial for generalization. For more details, we refer to our rebuttal to reviewer 48oZ.

We are happy to include these detailed results.

# Samples vs FID on InteriorVerse

# Samples vs FID on G-Buffer ObjaVerse

FID Metrics

LoRA Rank

We explored different LoRA ranks and noticed differences in (a) image quality and (b) generalizability to out-of-distribution samples. A too low LoRA rank fails to produce high-quality images from the target domain. In contrast, a too high LoRA rank negatively impacts generalizability in a similar fashion as our analysis of dataset size for LoRA finetuning (Fig.9/Tab.2). Thus, we chose rank 64 as the best middle-ground that produces high-quality images and still generalizes well to out-of-distribution domains. We'll include these findings into our Fig.9.

Motivation for Image-Space PBR Priors

Text-based generative models can act as a generic prior over images, which has shown great value for numerous downstream tasks: improved intrinsic image decomposition(IID $29$ , RGBX $67$ ), depth estimation (Marigold $Ke et al CVPR 2024$ ), etc. Our goal is to establish a similar generic prior for PBR materials to support downstream applications like UV-based texturing.

The key challenge of the field of PBR generation is the lack of large-scale, diverse datasets of PBR UV textures. Unlike RGB data, material datasets are scarce, and crucially, consistent UV parametrization (mapping from object coordinates to texture coordinates) across assets practically does not exist. This makes training direct generative models in UV texture space infeasible.

Instead, we learn a prior in image space. To distill image-space information into texture-space, we use SDS, which aggregates image-space signals from multiple views. This enables PBR UV texture generation for 3D assets, even though trained with relatively small amounts of image-space PBR data. In contrast, decomposition methods require an image as input and it is not obvious how one would synthesize consistent PBR UV textures for novel 3D assets.

Accurate Reflections and Normal Maps

We propose an image-space rendering loss with our lighting direction sampling to improve the specular reflections. We show in Fig. 8 that without this rendering loss all modalities are averaged and washed away. The rendering loss helps to ground the individual PBR properties and leads to more details. Our lighting direction sampling further enhances this effect by strengthening the gradient signal coming from the specular highlights. Compared to the baselines (Fig. 6), our method exhibits much more details in the rough and metal maps, which ultimately facilitates higher-quality renderings. That being that we fully agree that there is still room for further improvements (e.g., with more diverse datasets); however, at the same time we argue that our approach makes a significant step forward.

Implementation Details

We curate 20 views from the InteriorVerse dataset across different scenes. Then, we select the corresponding albedo and normal maps for training our LoRA layers. Please also refer to the supplementary material L942 for more details.
Our cross-view attention mechanism utilizes the new LoRA layers inside of the attention blocks. Concretely, we add LoRA in every attention block for the to_q, to_k, to_v, to_out linear layers in our first stage training. In the second stage, we fine-tune these new layers with our cross-intrinsic attention. In both stages, we do not modify the pretrained prior weights, thus preserving the generalization capabilities. We are happy to elaborate further in the final revision.

We thank again for the reviewer's feedback and hope our rebuttal addressed their questions. Please let us know if there are still open concerns.