LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models

Thanks for your valuable comments.

Question 1: About the issue of understanding cases of error propagation.

Response 1: We have updated the example diagram of error propagation (Appendix A, Figure 4, page 14) in the updated version of the PDF. You can also click on this link: https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/error_propagation.pdf to view it on the anonymous website. ( If a PDF rendering error occurs, manually refreshing the page can resolve it )

Specifically, possible error propagation is that the history layout generated by the AR model contains an unreasonable Layout, which will affect the layout generation of subsequent elements, such as overlapping, unreasonable size/location, etc. However, these all happen under the condition of insufficient training. All models in this paper have been fully trained, so the impact of this error propagation is minimal.

Firstly, thank you for your detailed suggestions and constructive comments.

Regarding the issue you mentioned in Weakness 2 (about the lack of unconditional generation results in the paper), we have provided a unified response in the General Response. Also, in the updated version of the paper, we have discussed in detail the effects of LayoutNUWA on unconditional generation in Appendix E (Page 16~17). Below is the experimental results:

Question 1: About Equation Issue

Response 1: In Equation 3, $M_{j}^{(t)}$represents the mask corresponding to the $j$-th element in a Layout for the $t$-th task.

Question 2: About the performance of LayoutNUWA on different datasets.

Response 2: This is a very astute observation. We will address your confusion from two aspects:

1. The RICO and PublayNet datasets are large. In previous works, even with not very large model parameters, the models perform well when trained on a large training dataset and then validated on a test set with a similar distribution. However, for datasets with less data, such as Magazine, the performance is poor. Therefore, our model improves more on the Magazine dataset. However, we can't specifically look at the improvement on a certain dataset. Specifically, we conducted Domain-Agnostic experimental settings, and LayoutNUWA has significant improvements on all datasets under this setting, which previous work could not achieve.

2. One of the success points of LayoutNUWA's good performance is that it uses the prior Code knowledge of LLM (Section 5.2, page 9). It's not surprising that it performs best on low-resource datasets (Magazine), which also proves our point that LayoutNUWA makes full use of the prior code and layout knowledge of LLM.

Thanks for your patient review.

Question 1: about the issue of input condition containing layout size.

Response 1: That’s a good question. In previous papers (including LayoutDM), a specific template containing the layout size information is provided during the rendering process. During the training process of the model, although the layout size information is not integrated, some restrictions are applied to the data, such as encoding relative position/size information (in decimal form) instead of absolute position/size encoding, which also indirectly incorporates the predetermined layout size information.

For LayoutNUWA, we believe that providing layout size information has several benefits:

1. The results generated by the model can be directly rendered, so layout size information is required;

2. We use absolute position/size encoding, which is closer to the real scenario and utilizes the code prior knowledge of LLM. Compared with relative position encoding, the model's generated results are more interpretable, so a layout size is needed as a prior condition to constrain absolute position/size encoding, which actually makes higher demands on the model's generation, i.e., modeling with absolute value is much harder than the relative value.

3. Layout size also indirectly reflects domain characteristics. For example, the RICO dataset is a mobile UI layout, which requires typesetting in a tighter space (so there will be some overlap), while the Magazine dataset has a larger space, so typesetting has less overlapping. This is especially important for LLM to do domain-agnostic (DA) tasks.

Question 2: About the lack of training details.

Response 2: This is a great suggestion. We provide more details here: we deployed and trained the model based on the open-sourced LLM training framework LLaMA-X and HuggingFace. We optimized the training using DeepSpeed Zero3 technology. For the DS settings, we set the learning rate to 5e-5. For the DA settings, we set the learning rate to 5e-6 to prevent model explosion. We trained on 128 V100 units until the loss on the development set converged, at which point we stopped the training.

Note: For more details about training and deployment, please refer to the latest version of the paper (Appendix G, page 18).

Thank you for your constructive comments.

Question 1: About generation diversity and overfitting.

Response 1: This is a great question. We construct the same input and take the top 4 paths with the highest probabilities from the model output. Our model can generate diverse layouts in the position and size of each element while keeping the overall position of elements unchanged. You can see the generated cases on our provided anonymous website in  https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/README.md, and this phenomenon is quite obvious. In the paper, all real designs are sampled from the test and development datasets (which is completely consistent with previous work such as LayoutDM). To further address your concerns, in the updated version of the paper, we used the DocSim metric in Appendix D (page 16) to show the distribution of training data similar to the test set, which also proves that our model is not overfitting on some specific layout distributions.

Question 2: About unconditional generation.

Response 2: That's a great suggestion. In the updated paper, we also explored the unconditional generation in Appendix E (page 16~17). Here is the model performance:

Question 3: About the domain confusion issue.

Response 3: This is a great question, and we specifically discussed this issue in Appendix F (page17~18). As shown in the table below:

, where we design a toy experiment and find: 1)when given more prior layout information, such as in the C+ S → P setting, the model is less affected by domain confusion as the model tends to “resist” such domain confusion, i.e., The performance of LayoutNUWA is much worse when domain confusion setting is applied compared to when it is not used; 2) However, when there is a lack of layout prior conditions, such as in the C → S + P or Completion setting, the model is greatly affected by the domain confusion as such issue simply causes minor disturbances to the results, i.e., the model “yields to” the domain confusion under such circumstance.
Overall, under the DA setting, the model indeed experiences Domain Confusion problems (mainly depending on the amount of given prior knowledge), which is enlightening when using LLM for DA Layout modeling. For more details, please refer to Appendix F of this updated paper.

Thank you for addressing my concerns. I have raised my score from 6 to 8.

Dear Reviewers,

Thanks for your review. Considering the response deadline is approaching, we hope to have more discussions with you. The corresponding responses are already presented in each review's response box. In addition to this:

1. We have added more experimental results and updated the submitted PDF;
2. We have provided an anonymous website https://anonymous.4open.science/r/LayoutNUWA-ICLR2024-Anonymous-Supplementary-Materials/README.md showing more experimental results.

If you have any further inquiries or suggestions, please do not hesitate to reach out to us.