RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image

Dataset Generalization: The method heavily relies on the Carverse dataset, which may limit its applicability to other types of 3D objects. Exploring data augmentation techniques or constructing additional datasets for different object categories would enhance the model’s generalization capabilities. Model Efficiency: The current model, based on U-Net Transformer, incurs high training costs. Investigating more lightweight architectures or efficient training strategies could improve the model’s practicality. Detail Generation: While the method achieves impressive results for overall car structures, the generation of fine details remains limited. Incorporating techniques to better capture intricate features would be beneficial. Real-time Rendering: The differentiable rasterizer employed in the model compromises real-time performance. Exploring more efficient rendering methods would enable the model to be utilized in real-time applications. Model Interpretability: The black-box nature of the model hinders understanding its inner workings. Investigating methods to enhance model interpretability would provide valuable insights into the generation process and potential limitations.

1. The insertion effect in Figure 6 of the paper is not satisfactory; the reconstructed car appears artificial when placed into the scene. The authors need to explain the reasons behind this. I suspect it is due to the lack of reflective texture on the car and the absence of shadow modeling. While I can accept the lack of shadow modeling, the absence of reflective texture implies inaccurate material prediction. Is this due to the poor quality of the dataset created in the paper?
2. The core contribution of the paper is the prediction of relightable objects, but the experiments lack validation of relighting, including the accuracy of materials and relighting. The paper also does not compare with methods that do relightable object generation, such as UniDream. Without these experiments, it is impossible to see the effectiveness of the paper's method. Moreover, the paper lacks relevant visualization results.
3. The paper states that the feed-forward model outputs pixel-aligned Gaussian primitives. Does this refer to Gaussian maps? This is not clear. If it is Gaussian maps, Figure 3 only shows the Gaussian maps from a single perspective, and the paper does not mention whether it is multi-view. If it is only a single perspective, how can it represent a 360-degree object?
4. In Figure 3, the feed-forward model is referred to as the "Relightable 3D-GS Generative Model", but based on the text description of the paper, I do not see that this GS prediction model is a generative model. Why is it called a generative model in Figure 3? Or is it just a typo?
5. The technical contribution of the paper is relatively weak, mainly adding the prediction of relightable Gaussian primitives to LGM. I believe this is not sufficient for the ICLR conference.

1. Lack of Real Data Testing: The experiments were conducted mostly on synthetic data, the paper only presents two qualitative results on real data in section 4.3. The authors could conduct more experiments on real data, especially providing some quantitative results.
2. Insufficient Analysis on Viewpoint Consistency: The paper lacks an analysis of material consistency between generated multi-view images and the input image. Without ensuring consistency in material properties across views, the relightable 3D-GS model may have limited relevance, potentially restricting its applicability.
3. Moderate Novelty: The design of the relightable 3D Gaussian splatting model and the generative model shows limited innovation compared to existing methods. While performance improvements are achieved, the methodological design lacks significant originality.

For table 1, the use of terms like 'high' and 'low' for the quality and diversity metrics is very ambiguous and biased, especially when many of the datasets being compared are real-world datasets. From the images presented in the paper, I don't observe a significant quality improvement solely based on these visuals. However, I do recognize the added value that Carverse provides through its inclusion of normal and material information, which may not be available in the other datasets mentioned. That said, when comparing diversity and quality, datasets such as CarStudio visually appear much more diverse and of higher quality to me. Therefore, using 'high' to describe both Carverse and CarStudio can mislead readers into thinking their visual appearances in terms of diversity and quality is comparable, which doesn’t seem accurate. If we further consider the difference in instance numbers, CarStudio’s over 200k instances are much denser than the 1,000 instances introduced by Carverse.

The experimental section appears biased. If my understanding is correct, when the authors compared the RGM model’s ability to generate car models with other SOTA models, they used 50 car samples from their own Carverse dataset. However, only the RGM model was trained on the Carverse dataset, while the other models had no prior exposure to this dataset during training. This creates an inherent advantage for the RGM model, as it has a better understanding of the Carverse data distribution, which would inevitably lead to better quality metric scores compared to the other models.

The authors claim that the quality of existing synthetic car datasets is insufficient for modern real car rendering, but from visual inspection, I don’t see a significant quality improvement in Carverse compared to other synthetic datasets like Objaverse. In fact, in some instances, Objaverse’s car models even appear to have better visual quality. To truly demonstrate that Carverse is superior to previous datasets, a more convincing A/B test with identical conditions under RGM would be necessary.

I would also like to see more examples of scene insertion results to better assess how well the RGM model handles relighting. In Figure 6, both scenes shown are under bright daylight conditions. It would be more convincing to see examples in dimmer lighting environments to showcase how the relighting function performs under different lighting scenarios.