Intrinsic-ControlNet : A Generative Rendering Approach to Render Any Real and Unreal

• Evaluation. Lack of important baselines. This work focuses on the area of sim-to-real generation, but lacks discussion and comparison with many previous works. Specifically, one can render a synthetic image based on the intrinsic conditions with rendering engine, and then perform sim-to-real generation to translate the image to a realistic one. There are many works in this line such as [1]. I suggest the authors to compare with these methods and discuss the pros and cons.

[1] Enhancing Photorealism Enhancement. TPAMI. 2023.

• Speed. When we are talking about the rendering, one main concern is the rendering speed. Rendering engine supports real-time rendering. But the generative method is somehow slow.

• Interaction. Could you show some relighting results? Given the intrinsic, we often need to render the images under different illuminations. This also requires the model to correctly process occlusion and environmental interactions.

• Metrics. Using CLIP score for evaluation is not enough. The authors can consider FID, and aesthetic scores for better evaluation.

• Diversity. The paper shows results of simple scenes. How about complex scenes? Would the details make sense?

• According to ablation results in table 3, the proposed modules seem trivial. The differences in clip scores are only 0.01. More analysis are needed to validate the effectiveness of proposed designs.

• A bit bold claim: As stated in Section 3.1 (Problem Statement) as well as the introduction, the authors define their framework as generative renderer (Eq. 1). However, I found it to be a bit overclaim for the following reasons: 1) There is no qualitative result to show the 3D consistency / temporal consistency of the generated results; and 2) As specified in Eq. 1, the lighting condition seems to be the input to the model, however, I do not find any lighting condition fed into the model throughout the method design. Therefore, generative image stylization could be a better and more accurate terminology for the proposed method.
• Strong assumption: the proposed method assumes the known intrinsic of images. However, in practice, those intrinsic images are predicted from the pretrained model (Intrinsic Diffusion) which could lead to unexpected errors for an arbitrary in-the-wild image. Given the fact that the intrinsic decomposition of in-the-wild images is still a challenging task that is unsolved, I would not prefer a model that relies on accurate image intrinsic as input.
• Limited quality: Although the proposed method shows overall intrinsic-conditioned stylization results in Fig. 4, 6, 7, I do observe severe artifacts when zooming in. For example, the car wheel is distorted and the appearance of buildings/traffic lights is also distorted. This could be better illustrated if the authors could provide full-resolution results. As a “generative renderer”, people do not want the renderer to alter the original content they fed in.

1. A notable limitation of the Intrinsic-ControlNet approach is the inevitable domain gap that arises from the training data. Specifically, the network is only trained on appearance condition images predicted from real images using pretrained diffusion models, whereas in practical applications, it is expected to generate images conditioned on appearance intrinsic images from synthetic images. As discussed in lines 283-285, this domain gap can lead to degenerate results.
2. The generated images exhibit unauthentic characteristics. For instance, the lighting of inserted objects in Figure 7 appears unnatural. Furthermore, the second column of Figure 8 reveals misalignment between the generated image and the normal after editing. Additionally, the third column shows Intrinsic-ControlNet hallucinating an unrealistic light on the floor, which does not correspond to the ceiling lights. Moreover, despite the authors' claim that their method can eliminate environmental lighting effects, the resulting image in the fourth column still retains indirect lighting from the two lamps.
3. Unlike traditional rendering engines, the generation process of Intrinsic-ControlNet inherently involves a significant degree of randomness. Consequently, ensuring 3D consistency in the generated images becomes challenging, particularly when rendering 3D assets from a continuous rendering trajectory. This can result in a jittering effect in the rendering output.

1. Lack of Comparative Analysis with RGB↔X. The paper does not compare the proposed method to RGB↔X or a re-implemented version of it despite the similarity in task setting and both methods use generative framework. Additionally, the authors mention that "existing methods (e.g., RGB↔X) face challenges with multiple control conditions as inputs," but these challenges are not fully explained. This absence leaves the effectiveness of the multi-conditioned ControlNet uncertain without direct comparison to RGB↔X.

2. Questionable Evaluation Metrics and Comparisons

   1. Effectiveness of CLIP Metrics. In Table 1, this method outperforms ground truth (GT) in CLIP metrics, raising doubts about whether CLIP can reliably assess the quality of rendered images. CLIP primarily measures text-image alignment, which may not adequately capture "accuracy" or quality in rendered images.

   2. Comparison with Render Engines. The comparison with traditional render engines lacks clarity, especially regarding the environment maps used. One text prompt (e.g. "a sunny day") could have different and varied environment maps which play a crucial role in final rendered image. In the last column of Fig. 6, the render engine appears to use a high-contrast environment map, resulting in a "less realistic" outcome. Additionally, the first two columns in Fig. 6 show sunset clouds in render engine results, which influence CLIP similarity scores but are not included in the text prompt, potentially skewing comparisons.