Two Heads Are Better Than One: Exploiting Both Sequence and Graph Models in AMR-To-Text Generation

We thank the reviewer for the insightful feedback. Please find the answers to some specific questions below.

Can you make this paper more accessible to an audience that is not already familiar with AMR?

I found the paper unnecessarily hard to read. The lead sentence of the abstract doesn't make it clear that AMR is well-established in the literature, and many of the cited papers have hundreds/thousands of citations.

This submission does not make it as clear that AMR is an established concept in the literature. Nor is it as clear what the task is.

The submission has a short section (4.1 Datasets) with references to LDC. This made it clear that this is a standard problem in the literature. But by the time I get to the discussion of the dataset, it should be clear what the task is, and how much work there is on this task.

Thank you for pointing out that our paper is hard to read. In the first submitted version, we stated in the second sentence of the abstract that AMR-to-text is a well-studied task. In the revised version, we have added this point to the lead sentence.

In the first paragraph, we further clarified the definition of AMR in the revised version, explicitly stating that this is an established concept. Figure 1 illustrates an example of AMR.

In the second paragraph of the introduction, we introduced the AMR-to-text generation task more clearly in the revised version. We also use Figure 1 to illustrate an example of AMR-to-text generation, as previous studies have done. We then stated that our model DualGen is aimed to do AMR-to-text generation in the rest part of the Introduction more clearly.

In Section 2 "Related Work", we introduced two types of previous methods on AMR-to-text generation, introducing that AMR-to-text is a widely-studied problem and there are a number of existing works exploring this topic.

In response to your feedback, we have revised some expressions in the abstract and introduction to enhance clarity and better articulate the established task.

The first part of the title (two heads are better than one) suggests that this result might generalize beyond this specific case (ensembling of s2s and g2s on two AMR benchmarks). Is this paper limited to this specific case, or should it be of interest to a broader audience that may not be familiar with AMR?

Dual encoder-decoder models have been widely used in NLP, as we stated in Section 2 "Related Works". Our main contribution is to:

1. Demonstrate that this architecture is also effective for AMR-to-text, since no previous work has employed a dual encoder-decoder model in this task.
2. Propose an innovative method for "pretraining" GNN encoders within the dual encoder-decoder framework, since previous studies showed that the pretraining technique is effective for language generative models and no prior work has explored pretraining a GNN for language tasks.

AMR-to-text is a representative and valuable task for graph-to-text generation, and our method is useful in this setting. Because there is extensive work on dual encoder models consisting of un-pretrained GNN encoders, our method has the potential to be generalized beyond this specific case. We will try to add more experiments to prove that this method is also useful for other tasks later if our paper is accepted.

The analysis in section 4.5 is superficial and needs additional in-depth analysis. The authors can include additional quantitative analysis by analyzing the types of failures for each model.

We added Section 4.7 "Model Failure" to the revised pdf, examining the failures of three models: Guo et al. (2019) (g2s), Ribeiro et al. (2021a) (s2s), and our model. Ribeiro et al. (2021a) shares the same architecture and method as Bart, which is the same as our model without the graph encoders.

We consider entries with a BLEU score lower than 25 as failed cases. We analyzed the graph size (indicated by edge number and node number), reentrance node number, and graph depth for failed cases of three models. We presented the results in Table 3 in the revised pdf and analyzed the results in Section 4.7 "Model Failure".

References:

• Guo, Zhijiang, et al. (2019). Densely connected graph convolutional networks for graph-to-sequence learning. TACL. https://aclanthology.org/Q19-1019/

• Ribeiro, Leonardo FR, et al. (2020) Investigating pre-trained language models for graph-to-text generation. NLP4ConvAI. https://aclanthology.org/2021.nlp4convai-1.20/

Some parts of the paper are not clear. The paper fails to report the inter-annotator agreement for human evaluations. The paper also includes typos.

Thank you for pointing out the need to add necessary details for the human evaluation experiment. In response to your feedback, we have included additional information on human evaluations in Appendix B. We also have corrected typos.

Could the authors elaborate on how each encoder contributes to the overall performance of the model? Would the authors consider conducting an ablation study? Specifically, it would be insightful to demonstrate that the pretrained encoder effectively handles Out-Of-Domain (OOD) test cases, while the graph encoder adeptly captures structural information.

It would be better for authors to include an additional ablation study for the proposed method instead of only showing the final model. In this way, readers can have a better understanding of the contribution of each component.

In our first version, we did perform an ablation study. We apologize for not explicitly mentioning that one baseline is equivalent to an ablation study.

The ablation study involves three models besides DualGen: DualGen without graph encoders, DualGen without sequence encoders, and DualGen with un-pretrained graph encoders. We reported results for DualGen without graph encoders in the first version. We have included the outcomes for the other two variants in the revised version.

1. DualGen without graph encoders

In Table 2, the baseline "Ribeiro et al. (2021a)" involves training BART with only vocabulary modifications, which is the same as DualGen without graph encoders. This model is also used in Section 4.6 where we analyze the impact of graph complexity. We also verified the results by training DualGen without graph encoders using our own code, producing similar results to Ribeiro et al. (2021a). Our omission in clearly stating that "Ribeiro et al. (2021a)" is the same as our model without graph encoders was an oversight on our part.

2. DualGen without sequence encoders

The graph encoder in our model is designed not to take all information, but to focus on edge relations. The performance of DualGen without sequence encoders is worse (BLEU ~23) than the full model (BLEU ~48), aligning with our expectations.

3. DualGen with un-pretrained graph encoders

The graph encoder of DualGen is large. The AMR train sets are small, which are inadequate to train the GNN from scratch. We have tested DualGen with un-pretrained graph encoders, which means the GNN is trained from scratch while the rest parts use Bart parameters. The result shows that the model learns nothing (BLEU < 2). This is not surprising considering the model size.

In response to your feedback, we have modified sections 4.3 Compared Models and 4.5 Main Results to enhance clarity regarding Ribeiro et al. (2021a). Considering the limited information provided by the ablation study and the 9-page limit of ICLR, we included the ablation results, evaluated all by our own code rather than using results reported by Ribeiro et al. (2021a), in Appendix A in the revised pdf.

References:

• Ribeiro, Leonardo FR, et al. (2020) Investigating pre-trained language models for graph-to-text generation. NLP4ConvAI. https://aclanthology.org/2021.nlp4convai-1.20/

We thank the reviewer for the insightful feedback. Please find the answers to specific questions below.

The idea is a little bit incremental. The graph encoder is built upon Song et al., 2020. The idea of the dual encoder is also not new. For example, OntG-Bart (Sotudeh & Goharian, 2023) also uses a GNN and BART encoder-decoder framework to generate summarization. The dual encoder has also been applied to the multimodal domain (Li et al., 2021).

We agree that dual encoders - one decoder is a common approach in NLP. However, we want to emphasize that our paper goes beyond the conventional use of a dual-encoder framework.

In NLP, employing dual encoders with different architectures involves combining a pretrained Transformer-like sequence encoder with a non-pretrained encoder (e.g., GNN). This is also the case in OntG-Bart (Sotudeh & Goharian, 2023), with a pretrained BART encoder and a non-pretrained GAT encoder.

While GNNs are frequently employed in NLP dual encoder models, no prior work has focused on pretraining the GNN specifically for language-related tasks.

In our work, we introduce a distinctive approach by "pretraining" the GNN encoder. This involves using pretrained parameters from the BART encoder to initialize our specially designed GNN, which is different from the established dual-encoder paradigm. This initialization is possible because we intentionally designed our GNN to resemble the Transformer architecture.

Our work is significant because:

1. GNN is powerful in NLP tasks. However, there is no existing pretrained GNN for language tasks that provides a resource for direct use.
2. GNNs, unlike PLMs with Transformer architecture, are usually incompatible with being trained on a large corpus
3. As far as we know, no previous work has concentrated on pretraining the GNN component in dual-encoder models to enhance its effectiveness. This implies that our work has the potential to generalize to all Transformer-based dual encoder models in NLP where one of the encoders is a GNN.

In response to your feedback, we have revised the Abstract, 1 Introduction, 3.2 Model Architecture, and 5 Conclusion in our paper to explicitly articulate our contribution to "pretraining" a GNN for language tasks.

Reference:

• Sotudeh, S., & Goharian, N. (2023, August). OntG-Bart: Ontology-Infused Clinical Abstractive Summarization. In Proceedings of the ACM Symposium on Document Engineering 2023 (pp. 1-4). https://dl.acm.org/doi/abs/10.1145/3573128.3609346

The idea is a little bit incremental. The graph encoder is built upon Song et al., 2020.

Although we used the idea of vectorized structural information in Song et al., 2020, there are essential differences between our encoder and Song's:

1. We use a different method to generate node embeddings, utilizing Bart vocabulary and using the average value, rather than defining a special in-domain vocabulary for AMR
2. We use the Bart word embeddings and two additional linear projections for vectorized structural information, rather than using sequences of edges and specially defined tokens
3. We use a different attention mechanism, which resembles the original Transformer rather than Song et al., 2020.

These differences ensure that the architecture of DualGen's graph encoder can be initialized with Bart parameters, and the graph encoder can integrate with the other part of the model seamlessly.

Reference:

• Song, Linfeng, et al. (2020). Structural Information Preserving for Graph-to-Text Generation. ACL. https://aclanthology.org/2020.acl-main.712/

Thank you for your response to my questions! The authors have addressed most of my questions. I raised my score to 6. However, the addition of a non-pretrained GNN variant ablation would greatly enhance the authors' claim of novelty.

We thank the reviewer for the insightful feedback. Please find the answers to some specific questions below.

Actually, the multi-source structure (multiple encoders - one decoder) is a common approach in many NLP tasks. Even this paper does present some enhancements for the AMR-to-text generation task, the novelty of the proposed method appears to be somewhat constrained.

We agree that multiple encoders - one decoder is a common approach in NLP. However, we want to emphasize that our paper goes beyond the conventional use of a multi-encoder framework.

While GNNs are frequently employed in NLP multiple encoder models, no prior work has focused on pretraining the GNNs specifically for language-related tasks.

In our work, we introduce a distinctive approach by "pretraining" the GNN encoder. We use parameters from BART to initialize the special GNNs in DualGen. This is different from the previous multi-encoder paradigm.

Our work is significant because:

1. GNN is powerful in NLP tasks. However, there is no existing pretrained GNN for language tasks that provides a resource for direct use.
2. GNNs, unlike PLMs with Transformer architecture, are usually incompatible with being trained on a large corpus
3. As far as we know, no previous work has concentrated on pretraining the GNN component in multi-encoder models to enhance its effectiveness. This implies that our work has the potential to generalize to all Transformer-based multiple encoder models in NLP where one of the encoders is a GNN.

In response to your feedback, we have revised the Abstract, 1 Introduction, 3.2 Model Architecture, and 5 Conclusion in our paper to explicitly articulate our contribution to "pretraining" a GNN for language tasks.

A commendable aspect of this paper is the authors' comparison of performance with GPT-4. However, considering that GPT-4 is a general model, the comparison may not be entirely equitable. It would be more appropriate to utilize an open-source large language model (e.g., Llama) for the experiment. This is crucial to verify the effectiveness of the proposed method in the context of Large Language Models.

We believe GPT-4 outperforms all other unfine-tuned LLMs, so we didn't conduct experiments on other unfine-tuned models. As for fine-tuning open-source LLMs on AMR2text, we believe it's not a fair comparison considering the model size and training cost.

Our reason for comparing DualGen with LLMs is that using small models for AMR2text is still valuable when LLMs are easily accessible, because un-finetuned LLMs perform badly, and fine-tuning LLMs is expensive. For better model performances, fine-tuning an LLM with a higher cost can always be an option.

It's worth noting that the DualGen method can extend to LLMs as long as they are open-source and Transformer-based. We plan to add an experiment to fine-tune a dual-encoder Llama with LoRA if our paper is accepted. We apologize for not being able to complete this during this rebuttal session.

In response to your feedback, we have revised the part “4.9 Comparison With The Most Powerful PLMs” in our paper to state our opinions more clearly.