Enhancing Language Models with Idiomatic Reasoning

We really thank the reviewer for the valuable advice and suggestion.

Here we provide explanations to your questions.

1. which language the experiments were done: we perform our experiments on English.
2. the availability of the code: we would release our code upon acceptance.
3. general evaluations of the proposed model vs. the original T5 on non-idiomatic tasks: we provide the corresponding evaluations in the Section A.4 in the appendix. We show that injecting idiomatic ability does not take away our model’s prior ability with compositional expressions. We perform experiments on sentiment classification and paraphrase identification.
4. to what extent we can rely on a dataset like MAGPIE: While it is true that "catch a cold" involves a degree of compositionality since "cold" retains its standard meaning, it is widely regarded in idiomatic studies as a fixed expression that conveys a meaning not entirely predictable from its parts. This nuanced interpretation aligns with idiomaticity in the broader sense, as it represents a conventionalized phrase with figurative aspects, even if not fully idiomatic in the strictest sense. Regarding the example "looks like" and "temperature [...] drop," it is important to clarify our data augmentation strategy. Our multi-view data augmentation is designed to transform the same set of idiomatic expressions into multiple tasks, each emphasizing different aspects of idiomaticity, such as contextual ambiguity, non-compositionality, and usage patterns. This method allows the model to learn the nuances and contextual dependencies of idiomatic expressions effectively. While "looks like" might be considered idiomatic in some contexts and "temperature [...] drop" metaphorical, our focus is on systematically labeled idiomatic instances to avoid overgeneralization and ensure task-specific learning.

We really thank the reviewer for the valuable advice and suggestion.

R1: Our empirical results on diverse benchmarks, such as IMPLI, I2L, L2I, and FNB, demonstrate the efficacy of our approach. The model not only shows improved performance on tasks involving idioms seen during training but also on those involving unseen idioms. This is a strong indicator of the model’s generalizability and robustness in handling idiomatic expressions beyond the confines of the MAGPIE dataset. Additionally, our model surpasses various parameter-efficient fine-tuning baselines and outperforms larger models such as LlaMa2 and Mistral, which highlights the effectiveness of our data augmentation and training strategies​

R2: Thanks for pointing this out. 3M additional parameters were added. Besides, the parameters of Adapter and Lora are at par with our proposed model. We will provide the test dataset sizes in the final version.

R3: While our current submission primarily presents results on idiom-related tasks, we have conducted preliminary experiments on additional NLP tasks such as Idiomatic Expression Paraphrasing, Literal Expression Stylizing (L2I), sentiment analysis and paraphrase generation, which are indicative of text generation capabilities. These initial experiments have shown promising results, suggesting that our method retains its efficacy across different types of tasks. Due to space constraints, these results were not included in the main manuscript but can be provided as supplementary material upon request.

R4: Our choice to use the T5-large model was influenced by the constraints of computational resources. The T5-large model is already a well-established, high-capacity model that provides a robust baseline for evaluating the effectiveness of our proposed methodologies. While we acknowledge that evaluating on larger models like Llama and Mistral would be ideal, the computational resources required to perform such experiments were beyond our current capacity.

Thanks for the rebuttal. I keep my original rating.

We really thank the reviewer for the valuable advice and suggestion.

R1: Thanks a lot for your suggestion.

1. The core idea behind our meta-pretraining approach is to bridge the gap between pre-training and downstream tasks more effectively than standard fine-tuning. While standard fine-tuning adjusts a pre-trained model's parameters to fit a specific downstream task, it often does not account for the diverse nature and requirements of multiple downstream tasks, especially when dealing with idiomatic expressions. Meta-pretraining, in our context, is designed to prepare the model for a variety of downstream tasks by optimizing the initial parameter settings.
2. Meta-tasks are indeed distinct from both pre-training tasks and downstream tasks. In our framework, meta-tasks are specifically designed to encapsulate various idiomatic properties, such as idiom identification, definition generation, and idiom usage generation. These tasks are formulated to introduce idiomatic reasoning capabilities during the meta-pretraining phase. By learning through these diverse meta-tasks, the model develops a generalized idiomatic knowledge base, which helps alleviate the disconnect typically observed between pre-training and downstream fine-tuning.
3. Our approach alleviates the disconnect by integrating a multi-view data augmentation strategy combined with meta-learning principles. This strategy ensures that the model is pre-trained on various idiomatic scenarios, preparing it for a wide range of idiomatic contexts it might encounter in downstream tasks. During meta-pretraining, the model undergoes multiple iterations of learning and adaptation through these carefully crafted meta-tasks, thus reducing the gap between the pre-trained model and the requirements of downstream idiomatic tasks. We would make these clearer in the final version.

R2: Thanks a lot for pointing this out. The naming of parameters, particularly the "non-compositional layer," was chosen based on the specific functionality and the theoretical underpinnings of the layer. This layer is designed to specifically address the non-compositional nature of idiomatic expressions, which is a well-known challenge in natural language processing (NLP) due to the idiom's meaning not being directly inferred from its individual components.

Q1: Thanks a lot for your suggestion. We would make the corresponding modification in the final version.

We really thank the reviewer for the valuable advice and suggestion.

R1: Thanks for pointing this out. 3M additional parameters were added. Besides, the parameters of Adapter and Lora are at par with our proposed model

R2: Thanks a lot for your suggestion. We could provide some examples in the appendix in the final version.

R3: The T5-large model was chosen as the backbone PTLM due to its robust performance across a wide range of natural language processing tasks and its alignment with our multi-task pre-training methodology. As highlighted in our paper, T5-large serves as a representative model for evaluating the effectiveness of our proposed Idapter mechanism and multi-view data augmentation strategy. We acknowledge the importance of evaluating the applicability of Idapter across different language models (LMs). While our current work focuses on T5-large, our methodology is designed to be model-agnostic. This is evident from the architecture of Idapter, which leverages parameter-efficient fine-tuning (PEFT) techniques that are compatible with a variety of PTLMs. Specifically, the design of Idapter incorporates non-compositionality scaling and task-specific scaling layers that can be adapted to different LMs without significant modifications

R4: While Table 2 indicates that data augmentation significantly improves performance, it is important to understand that this enhancement is maximized in conjunction with the Idapter mechanism. The multi-view data augmentation method effectively generates diverse idiomatic data, but its full potential is harnessed when combined with Idapter's ability to process and learn from this augmented data in a parameter-efficient manner. This synergy is evident from the performance gains across various tasks when both strategies are used together, as shown in our ablation studies.

Q1: Although Idapter like adapter can be applied to other tasks as well, our original aim is to study idiomatic expression processing. Idiomatic expressions are inherently non-compositional, meaning their figurative meanings cannot be directly inferred from their individual components. This non-compositionality, combined with their contextual ambiguity and sparse representation in text corpora, makes idioms a particularly challenging and compelling domain for testing our method.

Q2&3: Thanks for your suggestion. We would make the corresponding modification in the final version.