Look, Compare and Draw: Differential Query Transformer for Automatic Oil Painting

1. The technical novelty is somewhat limited. The whole pipeline is quite similar to PaintTransformer [1]. The main difference to PaintTransformer is using the features of a differential image (between the current canvas and target image) as queries to the Transformer, to predict stroke parameters. Except for this design, the other parts are very similar to PaintTransformer (Detailed below).

The entire Methodology Section (Sec. 3) is quite similar to PaintTransformer, including the method design and the writing.

a) The self-supervised pipeline where the strokes are randomly synthesized in Sec. 3.1, is the same as the self-training pipeline in PaintTransformer[1] Sec. 3.1, but without any citation.

PaintTransformer[1]: “To train a Stroke Predictor, as shown in Fig. 2, we propose a novel self-training pipeline which utilizes randomly synthesized strokes.”

This paper: “We employ a self-supervised pipeline in which the current canvas and target images are constructed using randomly synthesized strokes, eliminating the need for real images during training”.

b) The stroke renderer in Sec. 3.2 is the same as previous methods, e.g., PaintTransformer[1], and CNP [2]. Moreover, the writing is highly similar.

CNP[2] Sec 3.4: “The strokes parameters are 𝑠 = {𝑥,𝑦,𝑤,h,𝜃,𝑟,𝑔,𝑏}, where (𝑥, 𝑦) indicate the coordinate of the stroke center, 𝑤, h are the width and height of the stroke, 𝜃 is the rotation angle and (𝑟,𝑔,𝑏) is the RGB color of the stroke.”

This paper Sec 3.2: “The strokes parameters are s = {x, y, h, w, θ, r, g, b}, where (x, y) denotes the coordinates of the stroke center, h, w represent the height and width of the stroke, θ is the rotation angle, and (r, g, b) indicates the RGB values of the stroke.”

c) The training loss in Sec. 3.4 are the simple combination of PaintTranformer[1] and CNP[2], but without any citation. Moreover, even the formulas are almost the same as previous papers. The loss terms in Eq. (3)(4)(5)(6)(7) are exactly the same as the loss terms in PaintTransformer[1] Eq. (7)(8)(10)(11)(13). The adversarial loss part (Eq. (9)), introducing WGAN-GP into the training, is the same as CNP[2] Eq. (5).

d) The inference pipeline in Sec. 3.5 is the same as PaintTransformer[1] Sec. 3.5, but without any citation. Again, writing is highly similar to PaintTransformer[1].

PaintTransformer[1] Sec 3.5: “To imitate a human painter, we devise a coarse-to-fine algorithm to generate painting results during inference…Target image and current canvas would be cut into several non-overlapping P × P patches before being sent to Stroke Predictor.”

This paper Sec 3.5: “To generate painting strokes that mimic human artists, we predict strokes in a coarse-to-fine manner during the inference process… Both the target image and the current canvas are uniformly divided into multiple patches with a size of P × P, which are then fed into our painting network for stroke prediction.”

e) The Sec. 3.3 is the only part that is slightly different from previous methods.

If we compare the Sec. 3.3 of this paper to Sec. 3.3 of PaintTransformer, this paper and PaintTransformer[1] both use a Transformer structure for stroke prediction. Both methods extract features for canvas and target image, use a Transformer encoder and a Transformer decoder, and use two MLPs to predict stroke parameters and stroke confidences.

The main difference is that, instead of only extracting features $F_c$, $F_t$ for the canvas $I_c$ and target image $I_t$, this paper additionally extracts feature $F_d$ for the differential image $I_d=I_t-I_c$, and the $F_d$ is transformed into query tokens for the Transformer Decoder, while in PaintTransformer, the query for Transformer Decoder is learnable. This difference is a slight change to the inputs of the Transformer.

2. The quantitative and qualitative comparison are not comprehensive. The authors only compare under the setting of 5000 strokes. However, it is necessary to test under different number of strokes. Other strokes number setting should also be done, e.g., 200, 500, 1000, 3000 strokes, like CNP [2].

3. From the qualitative comparisons in Fig. 3, the differences between the proposed method and other methods are not quite visible. However, in the results of MOS in the Tab. 2, the gap of MOS between different methods is quite big. The MOS results are inconsistent with the qualitative comparisons. It is suggested to explain this inconsistency, e.g., give the detailed information of the user study.

4. From the qualitative ablation study results in Fig.5, it is difficult to see the difference, especially the Fig. 5(f).

[1] Liu S, Lin T, He D, et al. Paint transformer: Feed forward neural painting with stroke prediction[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 6598-6607.

[2] Hu T, Yi R, Zhu H, et al. Stroke-based neural painting and stylization with dynamically predicted painting region[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 7470-7480.

• One of my concerns is the use of synthetic data and self-supervised pipeline (similar to paint-transformer paper) while encoding a human-like inductive bias into the painting process.

  • For instance, while the use of differential features in Fig. 1, point to the semantically meaningful regions during the painting process, the same is unclear when using a synthetic data and random brushstrokes when adopting a self-supervised training approach.
  • Do the authors have any intuition on the above?

• The problem of stroke-efficiency has also been studied in [1]. In particular, it proposes a simple gradient-optimization of the original stroke parameters in order to improve the stroke efficiency in a post-hoc manner.

  • While the original paper uses Bezier curve stroke representation and the current paper uses oil painting strokes, it would be interesting to see if the brushstroke regularization from [1] can be used to further improve stroke-efficiency of the painting process (Fig. 3).

• While a minor point, does the proposed approach generalize to other stroke representations as in SNP paper [2]?

References: [1] Intelli-Paint: Towards Developing Human-like Painting Agents, Singh et al., ECCV 2022 [2] Stylized neural painting, Zou et al, 2021

1. This paper indeed makes some beneficial improvements to the Paint transformer, but for me, the paper is too focused on craftsmanship. The main goal of the paper is to reproduce a painting as closely as possible, as shown in Figure 3. Although previous work has some flaws, it is difficult for this paper to make a significant difference in this dimension.

2. Some sections of this paper, such as the Adversarial Loss, seem to describe content that does not differ significantly from previous work in this field. This part should be appropriately simplified because it is not closely related to the main contributions of this paper.