MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting

We thank the reviewer for the comments and note that the main concern lies in the effectiveness of the Blinn-Phong model used in our rendering pipeline. Specifically, whether it remains necessary given the presence of an effective meta-learning framework.

We argue that both the meta-learning framework and the Blinn-Phong shading model play essential and complementary roles in MetaGS, and that their combination is key to achieving strong generalization under OOD lighting. Below, we provide further results and discussion to clarify the respective contributions of the meta-learning component and the Gaussian-Phong renderer.

1. Meta-Learning with Different Relighting Models

Can you still achieve comparably high OOD test performance if you used another rendering model from prior work? ...I'm curious if the meta learning itself is robust enough or it has to be coupled with this Blinn-Phong model.

To investigate the general applicability of our meta-learning method, we incorporate it into several alternative relighting frameworks across different model families—including both NeRF-based and 3DGS-based methods, and ranging from BRDF-based to fully implicit formulations:

• Meta-learning + NRHints (implicit geometry + implicit lighting): This method does not rely on any parametric BRDF model, instead modeling full light transport via an MLP.
• Meta-learning + Neural Relightable Gaussians$^{[1]}$ (explicit geometry + implicit lighting): Each Gaussian is assigned a learned feature vector decoded into color via an MLP (completely free of explicit shading assumptions). We apply meta-learning in both of its two stages.
• Meta-learning + GS$^3$ (explicit geometry + explicit lighting with MLPs): This method uses a simplified BRDF formulation, along with an MLP-based shadow refinement module, and a residual MLP to compensate for unmodeled lighting effects.

The PSNR results are summarized below (NaN indicates model failure during RNG’s second stage):

These results demonstrate that while meta-learning consistently improves OOD generalization across different rendering models, its effect is especially pronounced when combined with explicit geometric modeling and shading methods, and is most significant with our Gaussian-Phong formulation.

[1] Fan, Jiahui, et al. "Rng: Relightable neural gaussians." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

2. Why the Blinn-Phong Model Works Well with Meta-Learning

The reason being Blinn-Phong model is still quite restrictive in terms of shadow effect... I'm curious if Blinn-Phong model is a contribution or a limiting factor.

From the above results, we observe that while the meta-learning scheme is compatible with various model designs, its benefits are most pronounced when combined with the physically structured Blinn-Phong model. Below, we outline two key reasons why this combination performs best:

• Simplicity & stable convergence: Despite its simplicity, the Blinn-Phong model defines a low-dimensional parameter space (ambient, diffuse, specular) for meta-learning optimization. Since meta-learning aims to capture shared structures across tasks (e.g., different lighting conditions in our case), this compact parameterization facilitates smoother and more stable gradient flow during meta-training, resulting in faster convergence and improved generalization of the learned scene geometries.
• Straightforward inductive bias of disentanglement: The physically grounded formulation naturally encourages a separation of global scene representations (e.g., geometry, materials) from lighting-specific illumunation factors. Compared to neural-based alternatives such as Neural Relightable Gaussians (RNG), the Gaussian-Phong model injects an explicit physical prior into the rendering process, constraining illumination components to follow interpretable, physics-based rules. Rather than embedding all lighting effects within a high-dimensional latent space, our method promotes the disentanglement of shared global attributes (e.g., scene geometry and material properties) from lighting-specific factors (e.g., direction and intensity) during meta-training.

It is worth noting that the full potential of the Gaussian-Phong formulation is only unlocked through meta-learning. While its low-dimensional structure provides strong interpretability and regularization, it does come at the cost of expressiveness, as pointed out by the reviewer. When trained in isolation (see our ablation in Table 3, “w/o meta-learning”), the Phong model can indeed lead to suboptimal results. The bilevel meta-learning loop addresses this by validating each inner-loop update on held-out lighting directions, effectively turning the strong physical prior of Phong into a practical advantage in OOD generalization.

3. Summary: A Balanced Trade-Off

In summary, OOD relighting poses a fundamental trade-off between in-domain expressiveness and OOD generalization. Our experiments demonstrate that combining a lightweight, physically grounded rendering model with meta-learning strikes this balance well. While the proposed Gaussian-Phong model is not the most physically accurate model, it functions as both a reasonably expressive renderer and a compact, differentiable prior that facilitates robust generalization through meta-learning across diverse lighting conditions.

Dear Reviewer,

Thank you once again for reviewing our paper. We would greatly appreciate it if you could take a moment to review our feedback and newly added experiments, and let us know if any concerns remain.

Best regards,

The Authors

We appreciate the reviewer’s time and effort in reviewing our manuscript and have responded to the points raised as follows.

Q1. The adoption of parametric models, like Blinn-Phong model, will inevitably impose limitations on the representational capabilities of 3DGS.

We thank the reviewer for raising this question. This touches on a fundamental trade-off between expressiveness and generalization ability. While we agree with the reviewer that introducing parametric reflectance models like Blinn-Phong may limit the representational capabilities of 3DGS, our design choice is motivated by the need to improve generalization, particularly in OOD scenarios.

We fully acknowledge that 3DGS is, in theory, capable of modeling arbitrary appearance and reflectance. However, in practice, especially under limited and biased OLAT data, standard 3DGS comes with significant risk of overfitting to training light distributions and entangled specular/diffuse representations, as we observe in our baseline comparisons (e.g., GS$^3$, WildLight).

The goal of incorporating a simple parametric model like Blinn-Phong is not to improve the rendering quality of 3DGS under rich and fixed lighting conditions, but to introduce a lightweight, physically grounded inductive bias that encourages the disentanglement of ambient, diffuse, and specular reflections, thereby enabling more robust and interpretable modeling of light transport.

In OOD relighting tasks, generalization is often more crucial than raw representational power. Despite its simplicity, the Phong model offers physically meaningful priors that help constrain the learning space under sparse and varied lighting conditions. Importantly, this inductive bias synergizes with our meta-learning algorithm, promoting the separation of global scene factors (e.g., geometry and material properties) from lighting-specific variables (e.g., direction and intensity). This significantly enhances generalization performance, as demonstrated in our experiments.

In summary, while we agree that parametric models like Blinn-Phong impose certain representational constraints, we believe that in the context of OOD relighting, these constraints serve as useful inductive priors. Our work takes a first step toward integrating physically inspired models into 3DGS to improve generalization, and we see great potential in future work exploring more powerful models that retain these desirable physical properties.

Q2. The experimental results obtained from 3DGS exhibit unexpectedly and unreasonably poor performance. Notably, given that the output of 3DGS is in the format of OLAT, it is surprising that even a straightforward convolution operation applied to any novel environment map would yield a significantly superior outcome compared to the results presented in the paper.

We appreciate the reviewer's comments and are committed to addressing all concerns thoroughly. However, we would like to kindly ask for some clarification to ensure we fully understand the points raised.

Could you please specify which particular experiment or result you refer to when mentioning the "unexpectedly poor performance" of 3DGS? Additionally, we are unclear about the phrase "a straightforward convolution operation applied to any novel environment map". Could you please elaborate on what this refers to? For example, is this referencing a known baseline or a particular technique?

Your further explanation will greatly assist us in providing a precise and constructive response. We thank you in advance for your guidance. We are also happy to provide a more detailed explanation during the subsequent author-reviewer discussion phase.

Dear Reviewer,

Thank you once again for reviewing our paper. We would greatly appreciate it if you could take a moment to review our feedback, and let us know if any concerns remain. We would be happy to answer them before the end of the discussion period.

Best regards,

The Authors

We appreciate the reviewer’s insightful questions and provide our responses below.

Q1. Apply meta-learning to other relighting models.

Yes, the proposed meta-learning scheme is inherently applicable to other rendering frameworks. To evaluate its generality, we integrate it into several alternative relighting models across diverse families, including both NeRF-based and 3DGS-based methods, and spanning from BRDF-based to fully implicit formulations:

• Meta-learning + NRHints (implicit geometry + implicit lighting): This method does not rely on any parametric BRDF model, instead modeling full light transport via an MLP.
• Meta-learning + Neural Relightable Gaussians$^{[1]}$ (explicit geometry + implicit lighting): Each Gaussian is assigned a learned feature vector decoded into color via an MLP (completely free of explicit shading assumptions). We apply meta-learning in both of its two stages.
• Meta-learning + GS$^3$ (explicit geometry + explicit lighting with MLPs): This method uses a simplified BRDF formulation, along with an MLP-based shadow refinement module, and a residual MLP to compensate for unmodeled lighting effects.

The PSNR results are summarized below (NaN indicates model failure during RNG’s second stage):

These results show that while meta-learning consistently enhances OOD generalization across various rendering models, its benefits are most pronounced when paired with explicit geometric modeling and shading methods. Notably, the meta-learning approach yields the largest improvement on our Gaussian-Phong rendering model. This is because the Phong model’s simple yet physically grounded design introduces strong inductive biases that enable meta-learning to capture generalized scene structural information even under limited lighting conditions.

[1] Fan, Jiahui, et al. "Rng: Relightable neural gaussians." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

Q2. Is there a specific reason why CUDA was not used for performance optimization?

Thank you for your question. Our implementation is fully GPU-accelerated and leverages PyTorch's built-in CUDA operations throughout. We did consider writing custom CUDA kernels, but found that the built-in operations were already highly optimized and sufficient to meet our performance needs. Moreover, our rasterization pipeline builds directly upon the same CUDA-accelerated primitives used in 3DGS, which are sufficient to achieve efficient training. As such, introducing additional low-level kernels would have increased implementation complexity without yielding significant performance gains in our setting.

Dear Reviewer,

Thank you once again for reviewing our paper. We would greatly appreciate it if you could take a moment to review our feedback and newly added experiments, and let us know if any concerns remain.

Best regards,

The Authors

We thank the reviewer for the valuable comments and address them below.

Q1. The OOD problem here is largely artificial and not representative of typical real-world conditions.

We appreciate the reviewer's comment regarding the nature of the OOD lighting setup in our work. While we agree that some OLAT setups (e.g., using a handheld flashlight) may not always exhibit extreme OOD lighting conditions, we respectfully clarify that our OOD lighting definition is not artificial, but rather a practical abstraction of real-world limitations. We address this concern from three perspectives:

1. Real-world OLAT captures are often spatially constrained

Even with a mobile light source, physical constraints (e.g., tripod positioning, occlusions, user-defined capture paths) typically result in non-uniform and biased light distributions. For example, OLAT captures using smartphone flashlights (like in [5, 34]) or mounted light rigs are frequently constrained to a subset of hemisphere directions. Unlike controlled lab environments, real-world conditions often involve relighting from previously unobserved directions (e.g., when user illumination changes or devices rotate). In such cases, generalization to unseen or poorly sampled lighting directions becomes necessary, which is exactly the scenario our OOD setup simulates.

Therefore, our OOD lighting split is designed to test the generalization capability of relighting models, which is a crucial requirement for practical deployment. As shown in Fig. 1, 3, and 4, existing methods suffer noticeable degradation even under moderate lighting distribution shifts, while MetaGS generalizes well.

2. OOD OLAT is only a training/test protocol; Generalization is the goal

While we train on OLAT, our method evaluates under broader settings, including environment map relighting and camera-light-colocated setups. These results (Tab. 4, Fig. 6) demonstrate that our method's benefits are not limited to synthetic OOD cases but also transfer to practical test-time distributions beyond the training configuration.

3. Broader applications

To better simulate realistic lighting biases, we conduct additional experiments restricting training lights to a fixed elevation, mimicking a planar circular light path, while the testing lights are placed at different heights. We also evaluate it under OOD clustered-light and few-light training setups, as detailed in our Responses to Q2. These experiments further support the method's applicability beyond synthetic extremes.

Finally, we would like to discuss another practical scenario--OLAT relighting for dynamic scenes. In such settings, where a scene is captured by a single camera and illuminated by a single light source (common in casual or consumer-grade setups), the scene’s motion is typically non-reproducible, preventing the dense sampling of illumination conditions. As a result, illumination coverage is inherently sparse over time, introducing a severe OOD relighting problem. While our current work does not explicitly target dynamic scenes, this scenario highlights the real-world significance of our OOD formulation. We see this as a promising direction for future work.

We thank the reviewer again for raising this important concern and hope this clarifies our motivation and rationale behind the OOD setup.

Q2. (1) The paper doesn’t clarify how many distinct light positions are used during training. (2) To better assess OOD robustness, could the authors evaluate performance when training with very few or clustered lights?

1. Number of light positions

In our OLAT setup, each image features a distinct light position and a distinct camera viewpoint. Specifically, we use 500 images (i.e., 500 unique point-light positions) for each synthetic scene and 600 images for each real-world scene. These capture details are provided in Appendix B (Lines 404–406).

2. Training with few or clustered lights

To evaluate MetaGS under constrained illumination conditions, we additionally design three settings that vary in light distribution, coverage, and sparsity, simulating more realistic capture scenarios:

• Clustered Lights: We construct lighting configurations with 1, 2, or 3 clusters on the illumination hemisphere. Each configuration is evaluated under two distinct settings:
  • Interpolation lighting: The cluster centers of the training lights are uniformly distributed across the hemisphere; test lights are placed between clusters.
  • Extrapolation lighting: Training lights are sampled from one side of the hemisphere, with cluster centers uniformly distributed (e.g., for two clusters, their centers share the same elevation of $45^\circ$; for three clusters, they form a triangular configuration). Test lights are randomly sampled from the opposite side.
• Few Lights: We sample 3 training lights in this setting, each paired with 200 camera views (i.e., one-light–multi-camera). Training light positions match the cluster centers used in the Clustered Lights setting.
  • Interpolation: Training lights are uniformly distributed; test lights sampled from the full hemisphere.
  • Extrapolation: Training lights are confined to one side; test lights from the opposite side.
• Fixed-elevation Lights: All training lights are placed at a fixed elevation of $45^\circ$, uniformly sampled along a horizontal ring. Test lights are drawn elevation bands outside the training range (i.e., $\leq 35^\circ$ or $\geq 55^\circ$). This simulates fixed-height capture trajectories.

All settings (except Few Lights) use 500 training images and 100 test images rendered under OLAT conditions, with unique light-camera pairs. We report average results on three synthetic scenes (Blender-rendered) in terms of PSNR / SSIM / LPIPS.

Clustered Lights:

Few Lights & Fixed-elevation Lights:

Observations and insights: As shown in the above results, our method consistently achieves strong performance across various lighting settings. In contrast, both baselines degrade significantly under sparse or biased lighting (e.g., Single Cluster, Two Clusters, and Few Lights). In particular, NRHints fails to reconstruct the Ball scene under the Three Lights setting, producing a collapsed geometry, which is similar to the failure case described in the paper's OOD setting. These results demonstrate that MetaGS not only generalizes well to unseen lighting conditions, but also remains robust under extremely limited or clustered illumination. This highlights its practical applicability in real-world relighting scenarios where lighting is sparse, biased, or expensive to capture.

Q3. Quantitative result of envmap relighting.

We note that this setting poses a particularly challenging generalization task, as our model is trained exclusively on OLAT data without any access to global illumination or multi-light supervision. While environment map relighting is not the primary target of our design, the results still reflect the model’s capacity to generalize beyond the OLAT setting. To this end, we supplement the following experiments:

Data construction: Constructing ground truth via weighted combinations of real OLAT images often leads to errors, due to limited coverage of light directions and colors from importance-sampled environment maps. To ensure reliable evaluation, we generate envmap-illuminated scenes synthetically using Blender, and conduct this comparison only on synthetic scenes.

Baselines for comparison: Most OLAT relighting methods (e.g., GS$^3$, NRHints, WildLight) do not support environment relighting. Methods that do support it (e.g., BiGS, PRTGaussian) typically require additional supervision such as all-light-on captures or multi-camera, multi-light setups, which fall outside our assumed constraints. We therefore use Relightable 3DGS as the primary baseline, since it supports environment map rendering under comparable supervision.

As shown, our method significantly outperforms the baseline in SSIM and LPIPS, with smaller gains in PSNR. We attribute this to imperfect brightness alignment, since environment maps are not seen during training and their HDR intensity values are manually calibrated at test time, potentially introducing global color shifts. Nonetheless, the strong perceptual scores and structural consistency highlight the robustness and generalization capability of our method to novel lighting conditions.

Dear Reviewer,

Thank you once again for reviewing our paper. We would greatly appreciate it if you could take a moment to review our feedback and newly added experiments, and let us know if any concerns remain.

Best regards,

The Authors