SciPG: A New Benchmark and Approach for Layout-aware Scientific Poster Generation

Q1-a：Why design a BiLSTM layer？

R1-a: As shown in Table 2 of our paper, the input text is quite lengthy, with an average of over 10,000 tokens. However, a standard Transformer model can only process up to 512 tokens at a time. To address this limitation, we adopt a hierarchical framework for the semantic representations. In this framework, a Transformer model at the lower level learns sentence-level semantic representations, while a BiLSTM at the higher level captures the contextual semantic representations for the entire document. We have clarified this in our revised version.

In addition, we also investigate the contribution of LSTM in the ablation study. The experimental results are shown in the Table.

From the experimental results, we can observe that our MDE suffers the significant decrease in performance when removing LSTM. We have updated the experimental results in our uploaded revised version of our paper.

Q1-b：what does ‘interactive’ refer to here?

R1-b: Sorry for your confusions. Our ‘interactive’ process does not involve user participation. It refers to an iterative generation process, as opposed to generating all content at once. The entire process is similar to dialogue generation. Given an input list from the extractor, at each step, we automatically feed one element of the input into the generator, which then returns its predicted result. This process is repeated iteratively until the entire input list has been processed. The design motivation behind this approach is driven by two main factors: 1）The input list contains a large amount of text, and concatenating all of it would result in a lengthy input that the generator cannot process at once. 2）By feeding the input interactively, the generator can better learn the relationship between the current element and previously predicted elements, which enhances both layout prediction and paraphrasing content generation. We have clarified this in our revised version.

Q2-a:  The layout generation problem.

R2-a: Before we present our work, we have carefully examined the existing layout generation task, which differs significantly from the layout prediction subtask in scientific poster generation. The layout generation task typically involves predicting the position and category of graphic elements on an empty canvas, an image (e.g., advertising posters), or an image with text constraints. However, existing layout generation methods primarily focus on the element categories, generating layouts with different categories of elements without considering the actual textual or visual content within each element. Essentially, these methods create layout templates with different categories of bounding boxes, which can result in a misalignment between the predicted bounding boxes and the content inside them. For example, consider a case with 20 elements, including 15 text elements and 5 figures. While existing methods might generate 20 bounding boxes with visually coherent layouts based on element categories (e.g., text and figures), they fail to address which specific text or figure should be placed in which bounding box.

In contrast, in our scientific posters, each element contains not only category information but also specific semantic content, which is crucial for ensuring the semantic consistency and coherence of the poster. Existing layout generation methods are not well-suited to our scenario. To resolve this, we propose an interactive layout prediction approach that focuses not only on the element category but also on the semantic content within each element. Given the complexity and diversity of scientific poster layouts, constructing a new dataset based solely on our collection of scientific posters could be valuable for the existing layout generation task. We also use the current SOTA layout generation approach, namely LayoutDM[1], to generate the scientific poster template based on our dataset. Unfortunately, LayoutDM[1] performs not good, generating plenty of overlapped bounding boxes. However, this task would be a distinct task from our own scientific poster generation. We plan to further explore this direction in future work.

[1] LayoutDM: Discrete Diffusion Model for Controllable Layout Generation

R1-a:

The reason for choosing BiLSTM instead of Transformer is not convincing.

1. The Bart-like [1] model uses a Transformer Encoder and a Transformer Decoder, which is a very successful architecture across many tasks. The difference between the model proposed in this work and Bart-like model is that this work uses a BiLSTM instead of a Transformer Decoder. There should be reasons for not using Transformer Decoder like Bart. Note that I do not mean using the exact architecture as Bart. I mean the Transformer Decoder like Bart is also a potential choice.

2. The correct way of using Llama and Phi in the task proposed in this work is not using them as a representation learning method. Instead, they can be finetuned to directly predict the extractive score. First, as these models are pretrained with longer context and more data, they may provide stronger prior knowledge and lead to better performance. Second, this approach eliminates the need for an extra BiLSTM and training from scratch.

In summary, Bart-like architecture (Transformer Encoder and Decoder) and Llama and Phi-like architecture (Transformer Decoder) are powerful choices. Given their strong performance across many tasks, the use of a BiLSTM in this work requires further explanation and comparison with these models.

In your current feedback, do you mean BiLSTM is better at processing longer sequence than Transformer Decoder? Could you please provide related work as well as experimental results to demonstrate it?

[1] https://arxiv.org/abs/1910.13461

R2-a:

1. semantic content or visual semantic features.

PosterLlama uses DINO v2 and Adapter to encode the semantic content of visual elements (see Section 3.2 of PosterLlama).

2. which content belongs to which text element.

PosterLlama can achieve it with a very minimal revision in the prompt. As shown in Appendix F of PosterLlama, by putting text content in each <rect /> item, it can achieve which content belongs to which text element.

3. long sentences problem.

How is this problem addressed by the approach in this work? I observed lots of overlaps between textboxes in qualitive results.

Q1: Why the approaches that tackle content extraction and layout in isolation mentioned in the introduction and related work couldn't serve as baselines. Providing a simple end2end baseline would also have been insightful.

R1: Sorry for your confusions. As a novel task, there are few feasible approaches that can be directly adapted to our scenario. Previous research on scientific poster generation has primarily focused on either poster composition or content extraction. The former proposed a simple probabilistic graphical model to infer panel attributes, but it requires human annotation for the panels of a poster. In contrast, our new dataset does not include panel information. For content extraction, we compare our model against two additional baselines: NeuralExt [1] and MSMO [2]. NeuralExt is a neural extractive model designed to extract text, figures, and tables from a paper. MSMO is a multimodal attention model that jointly generates text and selects the most relevant image from multimodal input. We adapted MSMO for the content extraction task. The experimental results are presented in the following table.

Our model outperforms the baselines due to its hierarchical structure, which is better suited for handling long documents. In contrast, NeuralExt achieves the worst performance in image extraction, as it is a single-modal model that relies solely on figure and table captions to represent visual elements.

Regarding an end-to-end baseline, there are currently no feasible end-to-end approaches that can be adapted to our task. Jointly training a content extraction model and a layout generation model is intractable due to the challenges posed by handling multimodal long documents and generating complex layouts.

We have updated the experimental result of multimodal content extraction in our uploaded revised version of paper.

[1] Xu et al. Neural Content Extraction for Poster Generation of Scientific Papers [2] Zhu et al. MSMO: Multimodal Summarization with Multimodal Output.

Q2: Providing more examples in the appendix, content polishment and the term SciPG is not defined anywhere in the text.

R2: Thanks for your suggestions. More examples, the duplicated information and the lacked details on evaluation metrics have been clarified in our revised version of our paper. For SciPG, it denotes our benchmark. We have clarified its definition in our introduction.

Q3: ImgR and ImgP are not clearly defined.

R3: Sorry for your confusions. ImgR represents the recall of extracted image elements, calculated as $\frac{\text{The number of correct images}}{\text{The total number of ground truth images}}$. ImgP represents the precision of extracted image elements, calculated as $\frac{\text{The number of correct images}}{\text{The total number of extracted images}}$. We have clarified this in the revised version of our paper.

Q4: Adding vertical bars or cmidrule's in Table 2.

R4: Thanks for your suggestions. We have updated Table 2 in the revised version.

Q5: The BiLSTM on top of the Roberta embeddings.

R5: Sorry for your confusion. As shown in Table 2 of our paper, the input text is quite lengthy, with an average of over 10,000 tokens. However, a standard Transformer model (Roberta) can only process up to 512 tokens at a time. To address this limitation, we adopt a hierarchical framework for the semantic representations. In this framework, a Transformer model at the bottom level learns sentence-level semantic representations, while a BiLSTM at the higher level captures the contextual semantic representations for the entire document.

In addition, we also investigate the contribution of LSTM in the ablation study. The experimental results are shown in the Table.

From the experimental results, we can observe that our MDE suffers the significant decrease in performance when removing LSTM. We have updated the experimental results in our uploaded revised version of our paper.

I thank the authors for their response. The authors were able to address my points of concern and thus I am raising my scores. I still recommend increasing the sizes of the example posters in the appendix, perhaps by putting them on separate pages.

Q1: The research novelty is not particularly significant, as poster design has been extensively studied.

R1: To the best of our knowledge, automatic scientific poster generation remains an underexplored area. Existing layout generation tasks primarily focus on advertisement poster design for products, where the diversity and complexity of layouts are significantly lower than in our case. In contrast, for scientific posters, previous studies have largely focused on either content extraction or poster composition, often relying on small-scale datasets. The lack of large, publicly available datasets has further hindered progress in this field. Poster composition methods have typically employed simple probabilistic graphical models to infer panel attributes, but these approaches require human annotation of poster panels. On the other hand, content extraction methods rely on predefined templates to generate fixed-format posters, which limits both the flexibility and aesthetic appeal of the resulting scientific posters. To address these challenges, we are the first to construct a large-scale dataset and propose a layout-aware poster generation approach that enables more flexible and visually appealing poster designs.

Q2:The authors could better highlight the differences of our approach.

R2: The main contributions of our work are the release of a new dataset and the introduction of a baseline approach to advance research in scientific poster generation. In our proposed method, while the multimodal extractor builds on existing RoBERTa and BiLSTM models, we have carefully designed its structure to effectively handle both visual content and long documents. For the interactive generator, we introduce an adaptive memory mechanism to address the challenges of generating long target sequences. Additionally, we design a layout-oriented pretraining approach to enhance layout prediction performance. The effectiveness of both the adaptive memory mechanism and the pretraining approach is validated through our ablation study.

Q3: Only compare with one baseline.

R3: As a novel task, there are few feasible approaches that can be directly adapted to our scenario. Previous research on scientific poster generation has primarily focused on either poster composition or content extraction. The former proposed a simple probabilistic graphical model to infer panel attributes, but it requires human annotation for the panels of a poster. In contrast, our new dataset does not include panel information. For content extraction, we compare our model against two additional baselines: NeuralExt [1] and MSMO [2]. NeuralExt is a neural extractive model designed to extract text, figures, and tables from a paper. MSMO is a multimodal attention model that jointly generates text and selects the most relevant image from multimodal input. We adapted MSMO for the content extraction task. The experimental results are presented in the following table.

Our model outperforms the baselines due to its hierarchical structure, which is better suited for handling long documents. In contrast, NeuralExt achieves the worst performance in image extraction, as it is a single-modal model that relies solely on figure and table captions to represent visual elements. We have updated the experimental result of multimodal content extraction in our uploaded revised version of paper.
[1] Neural Content Extraction for Poster Generation of Scientific Papers [2] MSMO: Multimodal Summarization with Multimodal Output

Q4: The generated posters are still poor and not suitable for practical applications.

R4: We acknowledge that the proposed approach is still a step away from fully automated poster generation. Even state-of-the-art AI models, such as ChatGPT, can make errors when performing specific tasks. However, our method focuses on automatically generating editable draft posters from research papers, enabling researchers to make only minor adjustments to produce the final poster, thereby saving significant time. Scientific poster generation is inherently challenging, with several complexities, including diverse and intricate layouts, a large number of elements, multimodal processing, and handling lengthy inputs. To address these challenges, we aim to release a new dataset and introduce a baseline approach to advance research in the field of scientific poster generation.

Q1: The large-scale dataset problem.

R1: Thank you for your suggestions. We have constructed a new dataset for scientific poster generation, consisting of over 10,000 (paper, poster) pairs, which is significantly larger than other existing datasets in this field. We have removed the qualifier 'Large-scale' in our revised version.

Q2: The other work: PosterLayout

R2: Thank you for your suggestions. PosterLayout focuses on advertisement poster design for products. Before presenting our work, we carefully examined the existing layout generation task, to which PosterLayout also belongs. This task typically involves predicting the position and category of graphic elements on an empty canvas, an image (e.g., advertising posters), or an image with text constraints. However, existing layout generation methods primarily focus on the element categories, generating layouts with different categories of elements without considering the actual textual or visual content within each element. Essentially, these methods create layout templates with different categories of bounding boxes, which can result in a misalignment between the predicted bounding boxes and the content inside them. For example, consider a case with 20 elements, including 15 text elements and 5 figures. While existing methods might generate 20 bounding boxes with visually coherent layouts based on element categories (e.g., text and figures), they fail to address which specific text or figure should be placed in which bounding box. In contrast, in our scientific posters, each element contains not only category information but also specific semantic content, which is crucial for ensuring the semantic consistency and coherence of the poster. Therefore, existing layout generation methods are insufficient for generating a feasible scientific poster.

We have clarified the difference between our work and the existing layout generation task in our related work.

Q3: Would it be possible to select a random layout and allow the baseline to generate its content? The same things applies the other way around, evaluating how the other baseline generates the layout.

R3:Thank you for your suggestions. In our scientific poster generation task, the layout of each element depends on its corresponding content. Adopting a completely isolated layout prediction approach is infeasible for generating a semantically coherent poster. For instance, a layout prediction model might generate a bounding box that is too small to accommodate a long sentence if it ignores the semantics of the given sentence. What you mentioned aligns with dividing our task into three subtasks: content extraction, content paraphrasing, and layout prediction. In fact, our baseline model, AdaD2P, follows such a three-phase approach for poster generation. The layout of element is based on the current context and the content within the element. However, our proposed method unifies content paraphrasing and layout prediction into a single subtask to better capture both the semantic representation within each element and the relationships among elements. In the evaluation, layout and content are also assessed separately in our paper.

Q4:  The texts are overlapping with each other in the qualitative samples 

R4: Thanks for your comments. We acknowledge that the proposed approach is still a step away from fully automated poster generation. Even state-of-the-art AI models, such as ChatGPT, can make errors when performing specific tasks. However, our method focuses on automatically generating editable draft posters from research papers, enabling researchers to make only minor adjustments to produce the final poster, thereby saving significant time. Scientific poster generation is inherently challenging, with several complexities, including diverse and intricate layouts, a large number of elements, multimodal processing, and handling lengthy inputs. To address these challenges, we aim to release a new dataset and introduce a baseline approach to advance research in the field of scientific poster generation.

Q5: Human evaluation.

R5: Thanks for your suggestions. In addition to the three existing human raters, we also invited seven additional raters to conduct the human evaluation. The results are updated in Figure 3 in our revised version of our paper. From Figure 3, we observe that the new evaluation results are on par with the previous ones.

Q6: The KL term in the generative loss.

R6: Sorry for the confusions. We include the KL-Divergence term in our training objectives because we observed that the training objective often suffers from overfitting, particularly in the bounding box predictions for layouts. To address this issue, we minimize the KL-Divergence between the softmax predictions (prediction distributions) and the one-hot output distribution (ground truth) with Label Smoothing. This approach helps mitigate the overfitting problem. We have clarified this in our revised version.