Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering

Thanks for your valuable review and suggestions. Below we provide a point-to-point response to all comments. If our response has addressed the concerns and brought new insights, we will highly appreciate it if the score can be raised.

Q1. One of the claimed novelty is the use of perspective projection with depth-aware rasterization. A similar perspective projection has already been discussed in 3Doodle (specifically in Appendix A.2), which seems to have no clear difference with the procedure proposed in this paper.

When transforming the 3D Bézier curve into 2D curves with perspective projection, the 2D projected curve is identical to a 2D rational Bézier curve defined by projected control points. To directly use DiffVG, which only supports ordinary Bézier curves, 3Doodle made a compromise by approximating the perspective projection as an orthographic projection. Specially, 3Doodle first projects control points to image plane with perspective projection, and then it draws 2D Bézier curve in image plane with the projected points, which approximates the rational Bézier curve.

To accurately formulate the differentiable rasterization of the projected 2D rational Bézier curves, we propose a novel rasterizer that supports 2D linear, quadratic, and cubic rational Bézier curves, which cannot be achieved by 3Doodle's ordinary (not rational) Bézier curve representation. We have comprehensive discussed the design details in Section 4.3 and provided theoretical derivations in Appendix C. Figure 2 has shown the relevant results, demonstrating further control over the shape of the curves by adjusting the weights of different control points.

Q2. While fixing occlusions in 3D curves is a positive addition to the rasterizer, there is no conclusive experimental evidence showing how this improvement impacts the quality of generation. For instance, it remains unclear whether depth-aware rasterization significantly enhances the final results beyond the occlusion analysis in Figure 11.

The proposed depth-aware rasterization can mitigate the noticeable depth or color conflicts which will disrupt the visual coherence of objects and lead to spatial and semantic ambiguity. We have conducted additional experiments to further evaluate its effectiveness, particularly on colored 3D sketch generation. Please refer to the Q2 of Common Response.

Q3. Applying SDS loss to rasterized images is relatively straightforward and does not present a significant technical difficulty, given its widespread use in NeRF-related tasks. This loss could be easily incorporated into prior approaches without altering their core components.

Although the SDS loss has been widely used in NeRF-related tasks, it has not been investigated in the 3D sketch generation task. Since the rendered sketch image is highly different from the conventional image and the sketch image is generated by the differentiable curve rendering process, the optimization is also challenging and it needs careful adjustments of the SDS components to guide the sketch generation.

We extensively discuss how the SDS components affect the 3D sketch results, such as the CFG weights and a biased sampling strategy, providing deep insights for future potential SDS-based sketch generation work, e.g., 3D sketch video generation and 3D sketch scene generation. We believe these technical designs and discussions will benefit the development of the community.

Q4. Moreover, the experiments do not convincingly demonstrate the advantage of direct SDS supervision on 3D curves. I think a better comparison would be another baseline like this: first train a 3D representation such as NeRF using the text2image or image2image SDS loss, then use the multi-view renderings of this representation to provide multi-view supervision for training other 3D sketch reconstruction methods, followed by a comparison in terms of quality and efficiency to truly demonstrate the benefits of directly optimizing 3D curves with SDS loss.

As suggested, we have compared our method to the pipeline mentioned above. Specifically, we used MVDream and Stable-Zero123 to generate 3D objects based on the input text and a single view image, rendering 120 views (view number similar to the NeRF Synthetic Dataset) as training data for the 3D sketch reconstruction method. Notably, we only report the comparison between Diff3DS and 3Doodle, as NEF struggled to converge and cannot achieve reasonable results in our experiments.

For the fair comparison, 3Doodle used the same setup as ours, i.e., it only optimizes the 3D curves with randomly initialized positions of the curve control points and does not utilize the prior from the point clouds obtained from the multi-view images. Also, the superquadric used by 3Doodle is not considered in this experiment. The results have provided in Appendix D.6 of the revised PDF.

We analyze the results from the perspectives of both quality and efficiency below.

Quality Comparison:

Overall, the quality of the sketches generated by the proposed pipeline and ours is comparable, indicating the SDS-based sketch generation can achieve similar qualitative performance comparing to the sketch reconstruction based on hundreds of multi-view images. Meanwhile, it should be noted due to time limit, the current comparison is not conducted on the colored sketch generation, which may reveal more differences for the two approaches, as shown in Figure 11 and Appendix D.7 of the revised PDF.

Efficiency Comparison:

1. For the Text-to-3D task, 3Doodle requires 3 hours (1 hour for 3D object generation and 2 hours for 3D sketch reconstruction), while our method takes 1 hour.
2. For the Image-to-3D task, 3Doodle requires 2.5 hours (0.5 hours for 3D object generation and 2 hours for 3D sketch reconstruction), while our method takes 2 hours.

Q5. In some examples, such as the phoenix, the 3D curves appear insufficient to fully capture the object's shape, and the curves still seem somewhat disorganized. Can this be solved with a higher number of 3D curves?

The primary reason for this issue is that the original SDS does not provide a strong structural similarity prior, in contrast to 3Doodle used losses like CLIP Loss and LPIPS Loss. However, for the 3D sketch generation task from text or single image input, since there is no multi-view ground truth supervision which can be used for the CLIP loss and LPIPS loss, we adopt the SDS loss for the curve optimization. The ablation study illustrated in Figure 9 (c) indicates that increasing the number of curves from 56 to 112 does not result in a more organized output. One promising solution could be to provide additional point cloud priors during initialization, similar to what 3Doodle does, by setting them as the positions of the control points.

Q6. The method initializes curves within a fixed-size sphere. How heavily does this initialization impact the final optimization?

Based on our observations, the quality of the results is generally insensitive to minor variations in the curve initialization radius. However, significantly decreasing the initialization radius can still degrade the quality of the results, as demonstrated in Appendix D.4 of the revised PDF. We recommend adjusting the radius to ensure the projected curve remains fully visible within the image and occupies a reasonable area.

Q7. Why not consider optimizing additional parameters, such as the width, color, or transparency of the 3D curves? Could optimizing these properties lead to significantly better results?

Our proposed differentiable rasterizer supports optimizing the control point position, color, opacity and curve width. To ensure a fair comparison, in our current experiments, we adhere to the baseline setup, focusing solely on optimizing the control point position. To address the concern, we conduct additional ablation studies to evaluate the effects of optimizing other parameters, as shown in Appendix D.5 of the revised PDF. It can be seen our pipeline can support the optimization of additional parameters including the color, opacity and width of the 3D curves. Meanwhile, we observe that optimizing the curve width may result in unstable outcomes. We hypothesize that this instability stems from the rendered image semantics being sensitive to variations in curve width, and plan to explore this issue in our future work.

I want to thank the authors for providing clarifications and more experiments about the major claims of the paper, which enhances the existing results. I am raising my score to 5.

Reasons for not a higher score: The added results still do not convincingly demonstrate the superiority of the paper's main claimed contribution—the improved rasterizer. The results in Appendix D.6 show only similar generation quality to the new baseline, with some improvements in efficiency. The benefits of rational Bézier curves and depth-awareness are minimal, even in the case of multi-view reconstruction. Moreover, the question of why the improved rasterizer is essential to the problem of sketch synthesis remains unanswered. This leaves the paper’s claimed contributions feeling somewhat fragmented. I do not mean to suggest that the improved rasterizer lacks value—it is indeed a meaningful addition to the current 3D sketch framework. However, the paper does not sufficiently explain why this component is critical for the task of 3D sketch generation.

Thanks for your valuable review and suggestions. Below we provide a point-to-point response to all comments. We hope our response has addressed the concerns and brought new insights.

Q1. I think 3D sketch/line can be an abstract representation of 3D structure, which can be an intermediate representation for some tasks like 3D reconstruction. In this senario, the differentiable rendering will also be very important. More discussions on this are strongly encouraged to make the application senarios more clear.

Thanks for the suggestions. We have included the following discussion in the introduction section of the revised PDF (L58-L61): "Moreover, in the area of wire art generation, 3D sketch or 3D curves are also widely studied to abstract the desired visual concepts from diverse inputs, e.g., 3D surfaces (Yang et al., 2021), multi-view images (Liu et al., 2017) and text (Tojo et al., 2024; Qu et al., 2024). These representations provide essential shape and structure of the artwork as well as hold significant potential as intermediate formats for tasks such as 3D reconstruction. However, generating view-consistent 3D sketches from flexible inputs, such as text or a single image, remains largely unexplored."

Q2. The authors is encouraged to present some numeric results of view-consistency.

Thanks for the suggestion. Currently, the view-consistency has been evaluated by the user study which asked the user to rate the four provided views based on the criteria including the 3D consistency. It can also be examined from the videos provided in the supplementary material. To further evaluate the view-consistency in a quantitative manner such as utilizing the optical flow to generate the warped images is very interesting and we will explore this direction in our future work.

Q3. It seems the number of curves is a parameter needs to be set manually. If it is so, please include this discussion into the limitation part.

Yes, the number of initialized curves needs to be specified by the user. We have provided the results of different initial curve numbers in Figure 9(c). Based on the experiments, we set the curve number to be 56 to achieve a balance between the approximation accuracy and complexity of the results. We have included the discussion on this limitation in the conclusion section of the revised PDF: "Also, the initial curve number is set manually to achieve a balance between the approximation accuracy and complexity of the results."

Thanks for your valuable review and suggestions. Below we provide a point-to-point response to all comments. If our response has addressed the concerns and brought new insights, we will highly appreciate it if the score can be raised.

Q1. The z-ordering of Bézier curves in Section 4.3 is the most technically advanced contribution. But the motivation is unclear: all sketches consist of black strokes, and intersections are rare at the image scale. Why care about correctly ordering the curves? Figure 11’s ablation shows only very minor visual differences. The effect of this ordering on 3D sketch generation is also left unclear. Why care about z-ordering of the curves? In which situation is it relevant?

First, we would like to clarify our motivation on introducing the depth-aware rasterization for rational Bézier curve rendering. Research on reconstructing or generating 3D sketches using multimodal pre-trained models is an emerging but underexplored area. The state-of-the-art method, 3Doodle, integrates DiffVG, a differentiable rasterizer originally designed for 2D vector graphics, as a component for rendering 3D curves. However, its performance is constrained by the inherent limitations of DiffVG's original design. To overcome this issue, we introduce a depth-aware rasterizer that enable precise and differentiable rendering of both black and colored 3D curves. We aim to advance research and foster further development in the direction of both 3D sketch reconstruction and generation.

In the current experimental setup, all curve colors are fixed to black to ensure fair comparison with baseline methods. Under this condition, depth-based rasterization indeed has minimal impact on the final results. On the other hand, z-ordering is crucial for more complex tasks like colored sketch generation. Accurate z-ordering ensures that occlusion relationships between curves are faithfully reproduced, enabling precise spatial and semantic representation. We conduct additional experiments to evaluate its effectiveness. Please refer to the Q2 of Common Response.

Q2. In the related works section, the motivations against existing 3Doodle approaches seem weak: Issues with perspective projections. Use of z-ordering for handling colored intersections.

Thanks for pointing out this issue. We will enhance the motivation against 3Doodle in the related work section. Specifically, in addition to the technical aspects like the perspective projection and z-ordering, we focus on using the simple text or single image input to achieve 3D sketch generation, while 3Doodle requires hundreds of rendered images for 3D sketch reconstruction. Although combining 3Doodle with some SOTA text-to-3D methods would achieve similar sketch generation results, our method is more efficient (as demonstrated in Appendix D.6 of the revised PDF), and it leverages end-to-end SDS-based differentiable curve optimization without relying on additional text-to-3D generation models.

Q3. The usefulness and relevance of 3D sketches as shape representation is not fully clarified. In which situation are 3D sketches useful?

Sketching is a powerful tool for visualizing concepts and ideas. 3D sketch which abstracts the 3D shape into 3D curves is a novel 3D representation that can provide more view-variant information than traditional 2D sketches. One key application of 3D sketches is the generation of multi-view consistent 2D sketches from different perspectives, facilitating a more comprehensive understanding of visual concepts. Additionally, this representation has great potential to serve as an intermediate representation for other tasks such as 3D reconstruction.

Q4. Figure 4: The meaning is unclear. Is this supposed to represent a pseudo-ground truth image? How does it relate to gradient supervision?

According to Equations 11 to 14, the SDS loss, which is conventionally formulated as the gradient form, can be interpreted as a scaled L2 loss that compares the rendered view with the pseudo-ground truth image predicted by the pre-trained text-to-image model. The differences are then used to update the 3D curve parameters through back-propagated gradients.

Figure 4 illustrates the pseudo-ground truth images under different levels of CFG weight, which is a key hyper-parameter in the SDS-based image or 3D generation. A larger weight provides a stronger shape supervision prior, while a smaller weight leads to shape degradation. The relevant ablation results are presented in Figure 9(a).

Thanks for your valuable review and suggestions. Below we provide a point-to-point response to all comments. If our response has addressed the concerns and brought new insights, we will highly appreciate it if the score can be raised.

Q1. The main contribution of this work appears to be the improved differentiable rasterization approach that enables more accurate depths during rasterization. However, it is not clear that this component is responsible for improved performance. Figure 11 ablates the impact of this rasterization technique on 3Doodle. However, the differences are quite minor and at this level of abstraction, the depths of these curves seem less important to the visual coherence of the object.

First, please refer to Q1 of Common Response for our main contributions. For the rasterizer part, our contributions include:

1. Extending DiffVG to support the differentiable rasterization of rational Bézier curves, enabling precise 3D curve perspective projection consistent with theoretical derivations.
2. Introducing depth-based rasterization, which facilitates accurate color computation and faithfully reproduces occlusion relationships between 3D curves in space.

In the current experimental setup, to ensure a fair comparison with baseline methods, all curve colors were fixed to black. Under this condition, depth-based rasterization had a minimal impact on the final results.

Nonetheless, our rasterizer shows considerable potential for tasks involving the rendering or generation of colored sketches. Notably, noticeable depth or color conflicts would disrupt the visual coherence of objects and lead to spatial and semantic ambiguity. We conduct additional experiments to evaluate its effectiveness. Please refer to the Q2 of Common Response.

Q2. The comparisons currently shown do not use strong enough / similar enough baselines. It feels that using score distillation guidance + DiffVG should still produce strong results for text/image guided curve generation. This is an important baseline and should be compared to in the paper.

As mentioned above, DiffVG, which is designed for differentiable rendering of 2D curves, is not inherently suitable for the theoretical framework of 3D curve rendering. A significant contribution of our work is the adaptation and extension of DiffVG to support tasks that require accurate perspective projection and occlusion handling for 3D curves.

To address the concern about the lack of sufficiently similar baselines, we add a new baseline method: first, we train a 3D representation use the SDS algorithm guided by text or image, and then use the multi-view renderings for training 3D sketch reconstruction methods.

We compare with the new baseline on both text-guided and image-guided 3D sketch generation, with results detailed in Appendix D.6 of the revised PDF. Across all tasks, we achieve performance comparable to the baseline method while considerably reducing the training time.

Q3. It appears from Figure 6 that, while your approach produces more 3D consistent results than 3Doodle, it less closely captures the structure of the guiding image. Do you have a sense of why that might be? Is it due to the fact that Diff3DS is more 3D consistent and thus is less able to overfit to exact structural details? Can this be improved?

The primary reason for this issue lies in the absence of a strong structural similarity prior in the original SDS method. In contrast, 3Doodle leverages loss functions such as CLIP Loss and LPIPS Loss, which effectively promote structural similarity. However, for the 3D sketch generation task from text or single image input, since there is no multi-view ground truth supervision which can be used for the CLIP loss and LPIPS loss, we adopt the SDS loss for the curve optimization. A potential solution could involve introducing additional point cloud priors during curve initialization, similar to 3Doodle’s approach, by setting them as control point positions. Moreover, as mentioned in the limitation part, distinguishing between view-independent curves (e.g., feature lines) and view-dependent curves (e.g., contours of smooth surface boundaries) could further enhance the expressive capacity of a 3D object’s overall shape.