Desigen: A Pipeline for Controllable Design Template Generation

Technical Novelty

The novelty of background generation is limited since the relationship between attention activation and subjects is explored in recent work Attend-and-Excite.

Though the similar analysis process, there are several key differences between our background generator and the recent work Attend-and-Excite:

• The motivation is different. Attend-and-Excite is a general method to solve catastrophic neglect (fails to generate one or more of the subjects from the input prompt) while our background generator aims at preserving space to overlay layouts for controllable design generation.

• The proposed method is different. Attend-and-Excite applies a Gaussian kernel on each attention map to obtain smoothed attention maps. Our method introduces the saliency map as an additional training constraint for background and reduces the activation of the corresponding region at inference.

• The applied stage is different. We propose salient attention constraint for training and attention reduction for inference while Attend-and-Excite is an inference-only method.

The layout generator is just a previously proposed technique, i.e., LayoutTransformer.

The key difference between our work and LayoutTransformer is that we enable the layout generator to be background-aware via a simple yet effective cross-attention mechanism. In the paper, Table 2 and Figure 6 show that the proposed layout generator outperforms LayoutTransformer by a large margin, as well as the VAE-based method CanvasVAE. This shows the proposed layout generator can serve as an effective baseline for future content-aware layout generation studies.

The proposed iterative inference strategy for background and layout refinement seems to be ad-hoc.

• It is natural for human designers to iteratively adjust the background and layout for better visual presentation. Compared with the single feed-forward generation, our iterative strategy allows the refinement between background and layout, leading to a more harmonious composition as shown in Table 3 and Figure 7.

• The idea of modeling the joint distribution of background and layout sounds interesting but much more challenging. For instance, a larger scale of design dataset and more computation resources are required for such joint modeling.

How is the attention map A obtained in Eq. (2)?

In the forward process of denoising U-Net, the cross-attention activations are calculated and stored for the salient loss computation.

Design Evaluation

The evaluation of the design is insufficient.

The numerical evaluation of the generated designs is an important but challenging task. In this paper, we try to build a comprehensive evaluation system for synthesized designs. We use 6 metrics to evaluate the design, including Salient Ratio (cleanliness), FID (aesthetics), and CLIP Score (text-image relevancy) for background, as well as Alignment, Overlap, and Occlusion to evaluate the layout and its combination with background.

Background diversity is not evaluated.

• Figure 11 of supplementary shows the background diversity given the same text prompt.

• Figure 5 of supplementary shows the background diversity given the same text prompt and different attention reduction masks.

Evaluation metrics of layouts are missing.

In Table 2, we have provided the typical metrics like overlapping and alignment, which are already enough to numerically show the quality of the generated layout.

The evaluation of the overall design is missing.

We use the Occlusion metric to evaluate the combination of background and layout, which can be regarded as an overall metric for design. This metric is also well aligned with human evaluation.

For the image-generation component, much of the same effect can be achieved with existing diffusion models by enhancing prompts like like a background image with empty space.

Although similar effect can be achieved with existing models, our background generator is more controllable and perform consistently well. To further demonstrate the effectiveness of our model, we show the baseline performance by enhancing context prompts with background-specific keywords (e.g., background image, with empty space to overlay layouts) with Stable Diffusion.

This table shows that, with enhanced prompts, the salient ratio is lower (which means there is more space preserved). However, the FID score becomes worse, which means the image quality may be damaged using such background-specific keywords. Instead, with the proposed method with only context prompts, both the salient ratio and FID become better, showing its effectiveness on background generation.

The layout-generation component of the work does not seem too novel.

• Designing a novel layout generator is not the main purpose of this paper. Instead, our key contribution to layout generation is to provide an iterative refinement strategy. Such an architecture-free strategy can further improve the visual harmony of the background and layout.

• The performance of the proposed layout generator is shown in Table 2, which indicates that such architecture is simple yet effective and works surprisingly well.

How is the prompt constructed in background generation experiments? Is it better to enhance the prompt with background-specific keywords for fair comparison?

For evaluation, we use the prompts from the test split of the proposed dataset. Most of these prompts only contain the context in the background (i.e., context prompt). To further demonstrate the effectiveness of the proposed background generator, we show the baseline performance by enhancing context prompts with background-specific keywords (e.g., background image, with empty space to overlay layouts) with Stable Diffusion.

This table shows that, with enhanced prompts, the salient ratio is lower (which means there is more space preserved). However, the FID score becomes worse, which means the image quality may be damaged using such background-specific keywords. Instead, with the proposed method with only context prompts, both the salient ratio and FID become better, showing its effectiveness on background generation.

Why the number of elements in a layout (and their minimal and maximal allowed size) is not an input to the background image generation model?

• First, it increases the ambiguity and difficulty of background modeling. The empty area in the background is not directly correlated with the number of elements, which makes the modeling more ambiguous. Furthermore, it is difficult to inject such low-frequent conditions (e.g., the number of elements) into the pre-trained image diffusion models.

• Second, it is not quite necessary. For the application of iterative refinement, if local editing is desired, a solution is to pass in an initial attention reduction mask for the background generator. Then the next-round refinement would be expected to be minor in this way.

How were the prompts collected in the Web-design dataset?

The prompts come from the alt text of background images and the description tag of the shopping webpage. We also parse some meaningful words from the background filename as a supplement. For more details of the prompts, Figure 10 in supplementary shows the word cloud of the top 200 words of the prompts on the proposed design dataset.

How to generate different ratios of backgrounds at inference?

Actually, Stable Diffusion can support various resolutions (although trained on fixed 512x512). We resize all the images (mostly in 3:4) in the dataset into 512x512 resolution for more efficient training. In inference, we generate 512x512 backgrounds and resize them back to 3:4 resolution. If other ratios of backgrounds are desired, users can just generate different resolutions of backgrounds and resize them correspondingly. Inference-only techniques like (Multidiffusion, ICML'23) can also be used to improve the image quality for such scenario.

How are the elements in each layout determined?

There are two ways to determine the number of elements:

• Decided by layout generator: Since the layout tokens are generated via an autoregressive Transformer, they reach their end when a <BOS> token is generated.

• Decided by user input: Each layout element is formulated by 1 category token plus 4 position tokens. The number of layout elements can be controlled by providing the category tokens and only the position tokens are generated for each layout element.

In our experiments, we use the latter way to control the number of layout elements (the number and type of category tokens come from the test dataset).