Zero and Few-shot Semantic Parsing with Ambiguous Inputs

Thank you for your comments and questions about our work, and for your attention in reviewing it. We've attempted to address the comments and questions from your review below and in the updated paper.

Weaknesses:

• “I feel like the combination of ICL and very constrained templates makes their zero-shot setting very hard…” Thanks for making this point. The constrained nature of the templates is indeed a limitation of our approach – however, we would like to note that the use of constrained decoding does alleviate this concern somewhat. Because the model is constrained to use the grammar of our templates, it can only generate syntatically-correct outputs. Thus, the model has help in generating valid templates, even if the template format is not what the model would naturally generate during open generation. Nevertheless, the point that open-source models struggle with generating correct LFs of either kind in the 0-shot setting is well-taken, and is an area for further research.

Questions:

• “In more real-world semantic parsing tasks there can be dramatically more than just two possible semantic parses. Would this approach (particularly use of EASL in collecting human judgements) still work in contexts with far more acceptable parses per sentence?” The human judgment collection would be substantially more difficult given >2 parses, but still possible. We could in principle use the same interface, and change the confidence scale. Right now, 0 confidence is translated to p=0.5 and 1 is mapped to p = 1. The logic behind this is that if the annotator were below p=0.5, they would have chosen the other option with some confidence > 0. This could be extended to multiple options, i.e. given 3 options, 0 confidence would mean p = 0.33. Alternatively, given multiple options, we could apply ranking-based approaches for data collection instead.
• “Insofar as all models were equally poor at a number of the zero-shot tasks at ZM_5, did the authors check the effect of raising that k parameter?” Due to computational constraints, we were unable to run higher batch sizes at scale. In our preliminary investigations, for a small subset of examples we found that the models we could run generally have poor beam diversity even with higher beam sizes, generally generating versions of the same output. After a certain point, the models tend to generate outputs that are structurally similar but have incorrect predicates in them, e.g. the predicate boy(x) might be used even though “boy” doesn’t appear in the input. This suggests that increasing the beam size is unlikely to change the ZM scores by very much in practice.
• “I was curious about the focus on capturing the training data biases rather than the human annotation judgements, for the few-shot setting.” Thanks for this point, it suggests an interesting direction for future work. In developing the FIM/FDM scores, we did consider whether to compare against human judgements. We opted to instead compare the range of possible ratios for two reasons. Firstly, we found in our human judgements that ambiguity is highly variable between examples, meaning we would need large amounts of data to make an accuracy comparison. For example, in Fig. 6 (now Fig 5.) PP attachment ambiguity examples range from 0 to 1. Averaging this to say that annotators generally think they have P(LF0) = 0.5 would obscure this variation, so we’d need instance-level annotations for all our evaluation examples. Secondly, ambiguity may not be a fixed quantity: we found that annotators differ confidently in their interpretations. Additionally, as pointed out by reviewer PRrS, there might be cultural differences in interpretation. Thus a fixed sample of interpretations might not be valid across different groups of people. We opted for metrics that subsume a metric based on estimates of judgments. Because we range the percentage of examples in the prompt from 0-100% in FDM and FIM, the true interpretation ratio should be found somewhere on this range. For example, if 30% of people interpret an example as LF0 and 70% interpret it as LF1, then this should be captured by FDM/FIM, since they test for this ratio. Marginalizing across ratios allows us to be agnostic about what the “true” ratio might be. Future work could expand this to a finer granularity – we considered increments of 10%, but given more examples we could consider finer increments.

Thank you for your attention to our work and your thorough questions and feedback. We have made several changes to the paper, which we hope address your concerns and which we highlight below.

Weaknesses

1. “...Why not formulate this as a multi-choice problem?...” This is a good question, which we have sought to address further based on your feedback. The reason to explore this problem semantic parsing (as opposed to multiple choice or open-ended language generation) is because of semantic parsing’s real-world applications. Parsing is used directly in real-world systems that have real consequences (e.g. robots, digital assistants, etc.). Ideally, these systems would be able to operate in open-ended environments, i.e. environments where we cannot frame the problem as multiple choice QA (since we don’t always know the choices, or there are too many to enumerate). Thus, while we’re interested in model’s performance on ambiguous language generally, we focus specifically on semantic parsing because of its practical applications. We agree that the results in a multiple choice setting may be very different from what we have presented, but these results would have little bearing on semantic parsing. We have added further motivation for exploring semantic parsing specifically in the introduction and in the discussion of implications of our work in the discussion (highlighted in blue).
2. “Is there any quantitative analysis? What kind of errors does the model usually make?” Based on this helpful suggestion, we’ve added a qualitative analysis to the appendix describing typical errors the model makes for each ambiguity type. We find that the best open-source model makes different errors depending on the ambiguity class, but many errors involve omitting key predicates or quantifiers.
3. 1. “ Why not use the same zero-shot and few-shot metric since the output format is the same?” The expected/desired behavior in the zero and few-shot settings is different. In the zero-shot setting, we don't know what the "true" output distribution is, so the metric just looks at whether the model puts enough mass on both interpretations to generate them. In the few-shot setting, we're giving the model evidence of the expected distribution, so we have a reference point, and can quantify whether the model captures that distribution. Thus, we need different metrics for the two settings.
   2. "Why not use language interpretation instead of LF generations? Language interpretation is much more intuitive for the model than LF generations." This is certainly true. However, in addition to the reason mentioned above for multiple choice (we care about semantic parsing, as opposed to general language generation) evaluation is very hard for language interpretation, and generally requires human evaluation, while automated evaluation for semantic parsing is easier since the form is constrained to a specific language.

Thank you for your thorough and insightful comments and questions. We have attempted to address most of the points made in our uploaded draft, where edits can be seen in blue. Below, we address each point in more detail.

Weaknesses

• “Human experiments…”

We are in agreement that the human results are imperfect, and have added a section on limitations of our human judgements to the appendix contrasting our study with standard methods in psycholinguistics and providing further justification for our choices. Regarding a comparison between human judgments to FIM: as-is, this judgment would not be a completely fair comparison, since human annotators were not shown any examples, while the model is shown 10 ICL examples with a particular ratio of LF1 to LF2.

• Assumptions about desired behavior

Thank you for pointing out this assumption. We do state the desired behavior of the model at the beginning of paragraph 3, but have added clarification about what the actual “model” is. As you astutely put it, having a distribution over interpretations makes sense if we are to “use the LM probabilities as a prior to guide some kind of search through the space of parses”. Since actual generations are being produced by beam search – which exactly is a search through the space of parses, guided by the LM’s probabilities – I think that we are in fact in agreement on the importance of the probabilities. We have added a point to introduction, clarifying that what is considered the “parser” is not only the LLM but also the search procedure used to actually decode parses (in our case, beam search). In terms of why we use beam search rather than sequentially decoding parses: we are aiming here to remain fairly close to how semantic parsing models are used in practice, rather than optimizing for performance on ambiguous examples specifically. We have clarified this in our limitations section.

• Guiding future research

Thanks for this feedback, which is shared with Q1 from reviewer PRrS. In light of this, we have updated the discussion section to include more direct recommendations for how we can improve semantic parsing under ambiguity (and why we might want to).

Questions/Typos

• “I think Figure 2 would be better as a grouped bar chart? It doesn't really make sense to me to have multiple models along a line here.” Based on your recommendation, we tried this plot style. We found that it was hard to parse which ambiguity type belonged to which model, since many of them (in fig. 2) are 0, so there are large gaps in the bar plot. These make it hard to determine which bars correspond to which model, which is clearer from the line plot.
• “"we rarely observe any models predicting both LFs correctly; one exception to this is conjunction ambiguity"— seems also true for bound?” Thanks for pointing this out – we have amended the text accordingly.
• We have clarified the point about semantic parsing being reduced to syntactic parsing. If there is a 1-to-1 mapping between syntax and meaning, then semantic parsing can be handled by a syntactic parser (this is of course not the case for real data).
• Re. the point on “multiple judgments”: Thank you for this observation, it is well-taken. We have amended the section to clarify that redundant annotations would only be required for the evaluation data, but that the train data requires diverse annotations. (see the added footnote 8)

We would like to thank the reviewer for their thorough review and attention to our work. We have uploaded a new draft of the paper (revisions in blue) where we have sought to address the weaknesses pointed out. We describe our changes/reasoning in more detail below:

Weaknesses

1. “[the paper] could benefit from more detailed explanation of how ambiguity affects real-world applications of semantic parsing.” We have added more details to the introduction, motivating why we examine semantic parsing and how misunderstandings could arise from ambiguity.
2. “The AMP dataset, while novel, might still be limited in scope and diversity, potentially affecting the robustness of the study’s conclusions.” This point is well-taken – while our study is an initial exploration into the underexplored area of ambiguity in semantic parsing, it is far from complete in terms of the range of possible ambiguities (and languages, settings, cultures, etc.) that could be explored. We have clarified our limitations subsection in Section 6 to emphasize that AMP can be extended both structurally (covering scope) and lexically (covering diversity).
3. “It is unclear how the proposed methods deal with the dynamic nature of conversational context, which can significantly affect ambiguity resolution.” Thank you for pointing this out – this is definitely a limitation of our approach, since conversational context often reduces ambiguity. We have highlighted where we state this assumption in the intro and have added it to the limitations. Removing context is realistic in the “digital assistant” domain, since most people would not accept a digital assistant that is constantly listening, i.e. by design, assistants have access to only parts of conversations. Note that some ambiguity is likely to persist even with context, as ambiguity commonly leads to breakdown in human conversation.