KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models

1. Although the proposed method balances the effects of coarse-grained and fine-grained features, it may not meet users' expectations when the input sketches are highly abstract. For example, as depicted in the last column of the second and third rows in Figure 4, the generated images align with the distortions shown in the sketches. Although this effect can be adjusted by the hyperparameter γ, changing the “controllable scale” within the ControlNet and T2I-adapter can also yield similar outcomes. To be specific, the authors are recommended to provide additional results based on varying the 'controllable scale' of ControlNet and T2I-adapter. This would illustrate that the proposed method can meet user needs without the need for adjusting γ.

2. Setting a hyperparameter γ to control the conditional signal based on the denoising steps during inference is a common operation in T2I task, which is not a novel technical contribution. The authors are suggested to demonstrate the differences between this method and previous ones. For instance, they could illustrate the relationship between the tanh-based modulator used in training and the γutilized during inference.

3. There are still some experimental issues:

(1). The evaluation of the generated results is somewhat subjective. Additional user studies could further demonstrate the strengths of the proposed method. Specifically, the authors can prepare more than 50 sets of sketches, which need to be evenly distributed across different levels of complexity. Users can then rank the results generated by the proposed method and other sketch-based methods.

(2). Since these models that assess user preferences are trained on their own designed datasets, a single aesthetic score is insufficient. The authors are suggested to conduct quantitative experiments using the HPS-v2[A] and Pick score[B] metrics, which are widely used for comprehensively assessing the generation quality.

[A]. Wu X, Hao Y, Sun K, et al. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis[J]. arXiv preprint arXiv:2306.09341, 2023.

[B]. Kirstain Y, Polyak A, Singer U, et al. Pick-a-pic: An open dataset of user preferences for text-to-image generation[J]. Advances in Neural Information Processing Systems, 2023, 36: 36652-36663.

(3). In the ablation study, there are too few examples, as the displayed images only include results of human portrait generation. The authors are recommended to add results of sketch generation for other categories. And the authors are recommended to provide more quantitative ablation results, with the same metrics as the quantitative experiments to demonstrate the effectiveness of the proposed method.

[-] The paper includes a qualitative and quantitative comparison with several SoTA methods. However, a thorough quantitative ablation study examining the impact of each component would greatly enhance the reader's understanding of their relative importance. There exists an ablative study but it appears to be superficial and does not justify all design choices made in the paper.

[-] The quantitative results provided in the work lacks critical information regarding the value of knob used -- necessary to gauge the improvement of the proposed model. It would have been great to see the changes quantitatively on how the actual numeric values of the knob affecting the final output metrics.

[-] Generally the contribution factor of both micro and macro pathways should sum to 1. However, in this paper, the knob during training increases the contribution of micro pathway in each epoch, without decreasing the macro pathway. Won't doing this cause issues in the overall magnitude of the feature map (as one pathway is getting more weightage than the other)?

[-] Any particular reasoning behind using a "tanh" instead of any other function in knob during training for increasing the contribution of micro pathway? It would have been great to see ablation in this aspect as well, to fully justify this design choice.

[-] Other SoTAs like ControlNet also provides a balancing factor to control sketch-conformity. The proposed method should be compared in this aspect as well to check how granular the control is compared to SoTAs.

[-] It would be very interesting to see "sketch-only" generation results, either by passing null-prompt or making the text-pathway weighting factor to zero.

• The professional sketches (Multi-Gen-20M) considered in this work are in binarised versions of HED edges, which is very different from what a real artist would draw (no artist or professional sketcher would produce lines like those in Figure 1). This makes the basic assumptions/conditions of the paper not very rigorous, somewhat deviating from the ambitious objectives, i.e., dealing with pro-sketch and any other complexity levels with a unified model.
• The modulator is heuristically designed. It is hard to justify if there is a scalability issue that might need tedious hyperparameter tuning for diverse training data.
• The effectiveness and applicability of the knob mechanism is questionable.
  • From Figure 6, the effect does not seem very pronounced: in the volcano example, the volcano corresponding to the intermediate gamma value appears to match the details of the input sketch better; in Keith's example (the second row from the bottom), the changes in facial details are also not noticeable.
  • Besides, the user has to try different knob values until satisfaction (and this may be pretty different for diverse input sketches) since it has no apparent relation to the user's need for the complexity level from the input sketches.
  • The impact of fine-grained cues is hard to manage precisely, as they have been injected into the model at early denoising steps, and the effect will last in the following denoising steps.
• The current competitors in experiments are not designed for sketches. It would be great if some sketch-guided image generation works, e.g., [a], could be compared and discussed.
• There is a “second evaluation set” with 100 hand-drawn images created by novice users used for experiments. It would be great to show these sketch images for completion.

[a] Sketch-Guided Text-to-Image Diffusion Models, SIGGRAPH 2023

1, I can not follow this paper. The writing is bad, since there are many sentences I can not understand. In introduction, the presentation is not good. (1) ‘Furthermore, we observed that the quality and alignment of the generated images with the input sketch are highly sensitive to the weighting parameter that governs the model’s dependence on the condition, Figure 2. (2) 'Additionally, the removal of the text-based conditioning in DM makes these models ignore the semantic power provided by text in diffusion models trained on large-scale image-text pairs, Additionally,...'. There are two ‘Additionally’ , which is not enjoyable. (3) 'Professional-oriented models like ControlNet and T2I-Adapter are designed to handle only artistic-grade sketches Fig. 3.a, while amateur-oriented approaches Koley et al. (2024), cater to novice sketches without text guidance Fig. 3.b'. In some places it is Figure , while some ones are Fig.. (4) 'The Macro Pathway extracts the high-level visual and language semantics from the sketch image and the text prompt using CLIP encoders and injects them into the DM via our proposed Coarse-Grained Controller (CGC)'. too much ‘ and’

2. Some figure captions lack clarity; for instance, in Figure 2, the term “weights” could use more explanation.

3. There are concerns that the overall writing style may resemble that produced by a language model, suggesting a need for a more personal touch.