Bridging semantics and pragmatics in information-theoretic emergent communication

We thank the reviewer for the positive and encouraging feedback!

To address the reviewer’s questions:

The paper's approach is interesting. To just confirm my understanding: in this specific setting, pragmatic representations will be always more specific (like hyponyms) than semantic representations, rather than dealing with connotations. This seems similar to the challenge of finding the most accurate hyponyms to describe something. Is my understanding correct?

This is partially correct. While agents could learn the strategy described by the reviewer, they may also learn a mix of strategies that humans employ. For example, a related phenomenon is known as scalar implicatures, in which a weaker characterization, such as ‘dog’, may implicitly rule out references that correspond to stronger characterizations, such as ‘Pomeranian’ (imagine a scene with a Labrador and a Pomeranian; both are types of dogs but Labrador is considered a more typical dog compared to Pomeranian). Humans also employ strategies with non-hierarchical alternatives (Silberer et al., 2020). That is, different conceptualizations for the same object can be produced by speakers to highlight different aspects of the referent. For example, we can call someone “teacher” irrespectively of any context, but also refer to them as “tennis player” when the context requires us to do so, even though “teacher” and “tennis player” are not in a hypernym-hyponym relationship. Our agents may also converge on solutions of this kind.

“What do you think would be the potential real-world applications of this work?”

While our work focuses on the science of intelligence, rather than on engineering AI applications, we see at least one important direction in which our work could potentially lead to improvement in real-world applications. Recall that our agents are trained without any human-generated linguistic data. This stands in sharp contrast to LLMs that require massive amounts of training data, which is unavailable in low-resource languages. Therefore, our work could help understand how to leverage cognitively-motivated optimization principles for training more data-efficient language models, with the hope of advancing more equitable language technologies. We thank the reviewer for raising this question and we will address this point in the paper.

References

Silberer, C., Zarrieß, S., and Boleda, G. (2020). Object naming in language and vision: A survey and a new dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5792–5801, Marseille, France. European Language Resources Association.

The explanation for the significance of the scenario that training in a pragmatic setting and testing in a semantic setting is insufficient.

What is the significance of training in a pragmatic setting and testing in a semantic setting?

We believe that this concern is somewhat related to the framing concern of reviewer Cm9N. As we explain in the related work section, and in our response to Cm9N, semantics and pragmatics are two key aspects of linguistic meaning that have so far been mostly studied separately from each other. While they are related, the interface between them has been largely under-explored and is not well understood. Most computational models of pragmatics assume a given lexicon, and most of the work of lexical semantics ignores the context-sensitive aspect of meaning that is captured by pragmatics. As we note in the abstract: “we aim to bridge this gap by studying how a shared lexicon may emerge from local pragmatic interactions.”

We believe that clarifying the framing throughout the paper, as we explain in the global rebuttal, will address this concern.

The paper's contributions are somewhat unclear.

Our contributions are stated in the abstract (lines 9-15), introduction (lines 68-75), and conclusion section (lines 294-304). We would appreciate it if the reviewer could explain what is unclear about these lines in the paper, so that we could address this concern. Otherwise, this comment by the reviewer is too vague.

Except for the point that the communication model is trained in a pragmatic setting, it is similar to [1] and [2]. It uses VQ-VIB as presented in [1], and the overall communication model is similar to [2].

While our model builds on the work by Tucker et al. that the reviewer mentioned, we extend that work in several important ways. First, not only is our training procedure different, but also our communication model is different. To see this, compare Fig. 2 in our paper with Fig. 1 in each one of the papers the reviewer cites. The evident differences are crucial for our work: we adjust the model to incorporate a shared context for the speaker and listener and implement a masking mechanism that allows us to invoke the model in two different ways (pragmatics and semantics settings). Second, in contrast to Tucker et al., who considered a discrimination task between whole images, our task focuses on discrimination between co-occurring objects within single images. This is crucial in our case, as we aim to capture communication in naturalistic scenes. Finally, our work addresses a major open question that has not been addressed by Tucker et al. That is, we focus on the under-explored interface between semantics and pragmatics, while Tucker et al. have focused on establishing the utility-informativeness-complexity tradeoff, and on the generalization and translatability afforded by this new framework. Thanks to the reviewer’s comment, we have noticed that these important differences were not stated clearly enough in the related work section. We will adjust that section accordingly to address the reviewer’s point.

Since the training loss includes utility, informativeness, and complexity, isn't semantics considered during training?

The key point in this context is that semantics is not built into the agents and we do not assume that a shared lexicon between the speaker and listener is given a-priori. This is in contrast to the vast majority of studies in computational pragmatics that assume a given lexicon (for review see: Goodman, & Frank, (2016). Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences). Semantics is considered during training only implicitly, in the sense that we incorporate in our training objective a general optimality principle, the information bottleneck, that is believed to guide the evolution of human semantic systems.

[1] Dowty, D. R., Wall, R. E., & Peters, S. (1981). Introduction to Montague Semantics. Springer.

[2] Murphy, G. (2002). The Big Book of Concepts. The MIT Press.

[3] Rosch, E. (1975). Family resemblances: Studies in the internal structure of categories. Cognitive Psychology, 7(4), 573-605.

[4] Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., & Boyes-Braem, P. (1976). Basic objects in natural categories. Cognitive Psychology, 8(3), 382-439.

[5] Regier, T., Kemp, C., & Kay, P. (2015). Word meanings across languages support efficient communication. In B. MacWhinney & W. O'Grady (Eds.), The Handbook of Language Emergence (pp. 237-263). Wiley-Blackwell.

[6] Zaslavsky, N., Kemp, C., Regier, T., & Tishby, N. (2018). Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31), 7937-7942.

[7] ​​Zaslavsky, N., Regier, T., Tishby, N., & Kemp, C. (2019). Semantic categories of artifacts and animals reflect efficient coding. Proceedings of the Annual Meeting of the Cognitive Science Society, 41.

[8] Grice, Paul (1975). "Logic and conversation". In Cole, P.; Morgan, J. (eds.). Syntax and semantics. Vol. 3: Speech acts. New York: Academic Press. pp. 41–58.

[9] Goodman, N., & Frank, M. (2016). Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20.

[10] Monroe, W., Hawkins, R. X. D., Goodman, N. D., & Potts, C. (2017). Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5, 325–338

[11] Franke, M. (2012). Signal to Act: Game Theory in Pragmatics. PhD thesis, University of Amsterdam.

Clarity + Questions (major)

The main goal of the paper is to study the interface between semantics and pragmatics, and specifically, to address the question: how can a shared human-like lexicon emerge from task-specific, context-sensitive pragmatic interactions? As the reviewer noted in the “Questions” section of their review, this framing is clear in the title and introduction (we also believe it is clear in the abstract, related work, and conclusions), but less so in other parts of the paper. We thank the reviewer for noticing this, and believe that this shortcoming is due to our focus on the technical details in the more technical parts of the paper, such as the experimental setup and results.

To address this issue, the reviewer suggested that “there need to be more explicit definitions of what semantics and pragmatics is, why the distinction is important, and what the empirical investigation is telling us about it”. We agree and are grateful for this suggestion. Here we address the reviewer’s questions, and in the global rebuttal we explain how we will revise the paper to address this concern.

The term lexical semantics refers to the study of word meanings [1, 2], independently of any specific conversational context. An example we mention in the paper (lines 23-25) is that we have a shared idea of what the word ‘player’ means, regardless of any context or scene in which this word might appear. The structure and evolution of word meanings has been an active research area in linguistics and cognitive science for several decades [3-5], and it has recently been characterized using the information-bottleneck (IB) complexity-informativeness tradeoff [6, 7]. However, the vast majority of studies in this area do not take into account how words are used in context, which can alter their meaning.

The term pragmatics refers to the study of how the local conversational context may alter meaning in real-time, as speakers and listeners reason about each other’s intentions and beliefs [8, 9]. For example, a speaker may choose either ‘player’ or ‘batter’ to describe the target person in the red bounding box in Fig. 1. In scenes with only one player, the more frequent word ‘player’ may be easier to produce, but in scenes with more than one player this word would not be informative about the speaker’s intended meaning. Therefore, if the speaker takes into account how a rational listener would interpret its words, then in the scene shown in Fig. 1 the speaker should use the word ‘batter’ instead of ‘player’. This type of reasoning, called pragmatic reasoning, has been widely studied both experimentally and with computational models that typically use tools from Bayesian inference and game theory [e.g., 9-11], that emphasizes only utility maximization. However, the vast majority of studies in this area assume that the lexicon is given and fixed.

The distinction between semantics and pragmatics is important for several reasons: First, they capture two different aspects of language: semantics captures how meaning is structured into words regardless of context, while pragmatics captures the context-sensitive meaning. Second, they capture two different time-scales: word meanings are stable building-blocks of language that change relatively slowly, while pragmatic reasoning applies in real-time, everyday language use. Finally, these two aspects of linguistic meaning constitute two subfields of linguistics, and therefore much of the relevant literature, including in computational linguistics and cognitive science, is organized around this distinction.

At the same time, it is widely agreed that semantics and pragmatics are related, as the reviewer noted as well; however the interface between them has been largely underexplored. As noted above, these two aspects of language are typically studied independently (but see our related work section for a few exceptions). As we state in the abstract, our computational and empirical investigation aims to bridge this gap in the literature, by providing a unified computational framework for studying semantics and pragmatics simultaneously. To this end, we extend a framework that integrates the IB complexity-informativeness tradeoff that characterizes lexical semantics with utility maximization that has been used to model pragmatic reasoning. In addition, in our setup, we do not assume that the lexicon is given a-priori, in contrast to most models of pragmatics, but rather investigate how it may naturally emerge from context-sensitive interactions. Our results show that in order to understand how a human-like lexicon may emerge from pragmatic interaction, it is crucial to consider all three terms in our objective function: utility, informativeness, and complexity.

Reusability (major)

To address the reviewer’s concern about the availability of our code, we made our code available by providing a link to an anonymized repository. Following the NeurIPS instructions we received, this link was posted in a separate comment to the AC. We included in the repository a demo notebook, useful for exploring the ManyNames dataset and identifying target and distractor objects in the images, which we hope will facilitate further extensions of our work.

Generalizability (minor)

We completely agree with the reviewer that our framework is general and could be used to study additional environments and open questions in the literature. We will happily add a short discussion on this in the paper.

Comments (minor)

Thanks for noting the typos, we will certainly fix them. We are also very excited about considering the Mandarin Chinese data in ManyNames in future work! We would have loved to do so in this paper, but given the scope of a single NeurIPS paper, we won’t be able to do justice to a cross-linguistic study in addition to our current set of results.

Empirical evidence suggesting that humans do not achieve near-perfect performance:

First, recall that our work focuses on understanding the emergence of the lexicon, and therefore our task setup only considers the use of lexicalized items (i.e., single signals or words) in communication. This kind of experimental paradigm is very common in the cognitive science of language, and is often used for studying the emergence of the lexicon. Second, while we do not have human data for our specific task, prior work has collected behavioral data for closely related tasks in this domain:

(a) The ManyNames dataset itself [9], which contains non-contextualized free naming data (i.e., with no distractor) from native English speakers, shows that the English lexicon contains unresolved ambiguities. This is not surprising given what we know about the lexicon and the probabilistic nature of many semantic categories [e.g., 1, 2, 5].

(b) Mädebach et al. (2022) [8] conducted a human experiment in a pragmatic referential task similar to ours (with distractors), but in contrast to our setup, they did not restrict participants to single-word responses. Their findings suggest that English speakers often cannot use the lexicon alone to resolve visual ambiguities in the ManyNames images. For example, the lexicon does not allow speakers to unambiguously distinguish between two similar chairs in the same scene. Instead, in such cases participants would need to employ syntactic constructions that go beyond the lexicon itself (e.g., “the chair on the left”).

While studying syntactic constructions is an important direction for future extensions of our work, our current paper focuses on how the semantic structure of the lexicon emerges, and this aspect of human language does not support perfect task performance.

On the importance of task performance vs other metrics for evaluation

“The utility-only objective achieves an accuracy of 95% but does so with a larger lexicon size, worse NID and differently looking complexity. So why do the complexity/lexicon size/NID metrics hold so much more value than the actual task performance?”

High task performance (utility) can be trivially achieved with very non-human-like communication systems. For example, by assigning a unique signal to each object in the dataset, or by using the same signal for entirely unrelated objects that never appear in the same context. Therefore, focusing only on task performance is unlikely to explain the emergence of human-like lexicons. Taken together with our previous point, that the English lexicon does not afford near-perfect task performance, we actually find it very encouraging that the emergent system that is most aligned with humans also achieves bounded task performance. In other words, we value all metrics for evaluation, including task-performance, and when comparing them with human behavior they all consistently support our conclusions.

References:

[1] Piantadosi, S.T., Tily, H.J., & Gibson, E. (2011). The communicative function of ambiguity in language. Cognition, 122, 280-291.

[2] Zaslavsky, N., Kemp, C., Regier, T., & Tishby, N. (2018). Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31), 7937-7942.

[3] Tucker, M., Levy, R., Shah, J., & Zaslavsky, N. (2022a). Trading off utility, informativeness, and complexity in emergent communication. Advances in Neural Information Processing Systems.

[4] Tucker, M., Levy, R., Shah, J., & Zaslavsky, N. (2022b). Generalization and translatability in emergent communication via informational constraints. In NeurIPS 2022 Workshop on Information-Theoretic Principles in Cognitive Systems.

[5] Malt, B. C. (1995). Category coherence in cross-cultural perspective. Cognitive Psychology, 29(2), 85–148.

[6] ​​Zaslavsky, N., Regier, T., Tishby, N., & Kemp, C. (2019). Semantic categories of artifacts and animals reflect efficient coding. Proceedings of the Annual Meeting of the Cognitive Science Society, 41

[7] He, Y., X. Liao, J. Liang, G. Boleda. 2023. The Impact of Familiarity on Naming Variation: A Study on Object Naming in Mandarin Chinese. CoNLL 2023.

[8] Mädebach, A., Torubarova, E., Gualdoni, E., & Boleda, G. (2022). Effects of task and visual context on referring expressions using natural scenes. In Proceedings of the Annual Meeting of the Cognitive Science Society, 44.

[9] Silberer, C., Zarrieß, S., and Boleda, G. (2020). Object naming in language and vision: A survey and a new dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5792–5801, Marseille, France. European Language Resources Association.

Weaknesses:

1. Major: The reviewer assumed that “people should be expected to achieve nearly 100% in performance” and based on that, argued that we should disqualify systems that do not achieve near-perfect task utility. We would like to clarify that the reviewer’s claim is inconsistent with prior work showing empirical evidence that people cannot achieve near-perfect performance in our task (see additional comment for detail). This also makes sense intuitively: our study focuses on the lexicon, and therefore our task focuses on the use of lexical items (i.e., single words/signals). Humans cannot achieve near-perfect performance with lexical items, but rather need to employ syntactic constructions for that. This stems from the ambiguity of the human lexicon, an important property that is believed to facilitate efficient communication [e.g., 1; 2]. Therefore, we find it encouraging that our approach has identified a regime in which human-like behavior arises both at the lexical level — as captured by NID, lexicon size, and informational complexity — as well as at the pragmatic level, as captured by the (expectedly bounded) task performance. We briefly explained this in lines 226-233 in our submission, but thanks to the reviewer’s thoughtful comments we realize that we should elaborate more on this point and explain it more clearly. We will certainly do this for the final version.

   Re the reviewer’s question on task performance vs other metrics: due to space limitation, we addressed this in the additional comment.

2. Minor: we will easily fix the typos and confusing enumeration. Furthermore, we will do a thorough English proofreading of the entire paper. The reviewer also noted that the labels in Fig2 are confusing, but they didn’t explain what is confusing about these labels (they seem rather intuitive to us). We therefore ask the reviewer for a clarification on this potential issue, to allow us to fully address all the concerns.

Questions:

1. Our chosen dataset of naturalistic images has several advantages: (a) richer lexicons, as the reviewer suggested; (b) using images of naturalistic scenes is particularly crucial in our case, because the structure of the emergent communication is sensitive to the statistics of co-occurring objects in the environment; (c) controllable environments have been considered in prior work with the utility-complexity-informativeness tradeoff [3], we thus extend this framework to more naturalistic settings;

2. Optimizing utility and optimizing informativeness are only partially aligned because they are different objectives that can lead to different solutions, but at the same time, they don’t necessarily compete and can sometimes support each other. For example, high informativeness can facilitate high utility, as seen in Table 1 ($\lambda_I = 1$). The intuition for this is that if the listener can accurately reconstruct the speaker’s representation of the target object, then it could perform reasonably well in many downstream tasks. At the same time, optimizing utility alone can be achieved with very poor feature reconstruction (i.e., low informativeness), as can be seen in Table 1 ($\lambda_U = 1$) and Figure 4a. The intuition for this is that when there is no pressure for informativeness, the listener can learn arbitrary representation spaces (e.g., random projections) that don’t resemble what the speaker has in mind. While this may yield good task performance for in-distribution inputs, it’s unlikely to generalize well to out-of-distribution (see [4] for preliminary results).

3. As noted above, our work focuses on lexical systems, i.e., the meanings of single words, and not syntax. We therefore allow our speaker agents to send only one word/signal per interaction. Respectively, the English ManyNames data that we use is also based on single-word responses from people. Our measures of complexity — the lexicon size and the informational complexity term — are both measures of lexical complexity, not syntactic complexity. The reviewer is correct that syntax is also an important feature of language; it is beyond the scope of this work and we will add a note on this matter in the limitation section.

4. This is best illustrated in Fig. 4a, which shows that in the utility-only condition the listener’s reconstructions are misaligned with the speaker’s representations of the target visual features, in contrast to other solutions (Fig. 4b and Fig. 4d) where the listener is better able to recover the speaker’s representations. We believe that the listener achieves high task performance in the utility-only condition because it learns random projections instead of useful reconstructions. The random representations could perform well for in-distribution inputs, but they are unlikely to generalize well to out-of-distribution inputs (see [4] for preliminary results).

5. The semantic setup is designed to recapitulate the human naming experiment with our artificial agents. In a naming experiment (such as the ManyNames experiment), participants are shown a single object and are asked to name it (using a single word). In contrast to the pragmatic task, there is no distractor and no downstream task. We use this setup to evaluate the emergent lexical systems with respect to the human naming data. NID, lexicon size, and complexity are all based on this setup.

Limitations:

Indeed, as the reviewer noted, this work doesn’t address the evolution of syntax. We focus on the evolution of the lexicon, which provides the building-blocks for syntax, and it is far from being well understood. We agree that this is a simplifying assumption and adding syntax is an important direction for future work. We will add a discussion on this in the limitations section.

As for the question about linguistic variation: it is well documented in the literature that languages vary widely in the ways they structure the environment into words, see for example [5-7].

Thank you for the details and clarifications. I have a few follow-up questions based on your response.

Firstly, could you quantify the expected human performance on the task and how you infer that number? I'm assuming it's definitely higher than 50% because it's unlikely that there are more than two objects with the same lexical label (on average). Is it closer to 60%, 70%, 80%, 90%, 95%? (To get further insights on the performance, could you maybe leverage the data split that Mädebach et al. 2022 introduced into no-competitor, lexicon-sufficient, and syntax-needed? I think this might potentially lead to interesting observations on error patterns and whether the cases where the model fails might be cases where people would employ syntactic specification.)

About the claim that syntax is excluded from this analysis: You specifically highlight that this work's contribution aims to only consider the semantics/pragmatics interface and tries to abstract away from syntax. I would like some clarifications on this point.

• Firstly, where do you draw the line between syntax and semantics in this particular setup? Mädebach et al. 2022 specifically discusses that they had to make some fairly ad-hoc decisions about what qualifies as lexical and syntactic modification. A prime example are compound nouns, where Mädebach et al. for instance chose a specific frequency threshold to determine whether they classify a noun compound as syntactic or lexical modification (making "tennis player" lexical specification and "front court player" syntactic specification). This is simply to illustrate that the syntax/semantic divide isn't as clear cut in this setting as you make it seem in your response and I would like some clarification on this point.
• Secondly, human language has developed alongside syntactic complexity, so the English language has fundamentally one major additional dimension of system complexity that actively negotiates semantic complexity. As you pointed out in your response, in Mädebach et al's work, they find that people choose syntactic structures over lexical specification quite frequently. The option to have syntactic variation at a speaker's disposal for a reference game or reference disambiguation task can then be reasonably expected to decrease the lexical variation that will emerge in a learning system. In fact, there is ample of evidence in the cognitive science literature that different languages indeed optimize for a distinct syntactic complexity/semantic complexity trade-off (see, e.g., Reali et al., 2018 [1], for an interesting exploration of this trade-off). And even in an already established language, like English, recent work suggests that people trade off lexical and syntactic complexity against each other in production (Rezaii et al., 2022 [2]). Given this, how are you thinking about comparing a system that fundamentally has one more dimension for reducing referential ambiguity to a system that is aimed to be stripped from it?
• This is a more minor point, but you highlight in your responses that you restricted the model learning setup to single words and that that prohibits learning of syntactic structures. Could you clarify how restricting an emerging language system to single words necessarily disables learning syntactic structure? Since the definition of what qualifies as a token is sort of arbitrary, couldn't the system in principle map to sophisticated morphosyntax? (To clarify, I don't believe that that's the case here but I would like clarification on how the single-word constraint relates to emerging-language syntax.)

Other points:

• You also stated that "High task performance (utility) can be trivially achieved with very non-human-like communication systems." -- I agree that utility is not a sufficient condition but what I raised in my review is the concern that a system that's not achieving task performance seems like it's missing a necessary condition to be an interpretable cognitive model.
• Do you have estimates on how stable the learned systems are and how generalizable the results? I'm aware that the tested model is a ResNet model which has been shown in the past to have promising representational alignment with humans. However, there are still many ad-hoc decisions that are part of training and evaluating such models as cognitive models which might significantly affect the results. For instance, is there evidence that the stopping criterion used for training is a reasonable stopping criterion for finding human alignment? If you were to continue training would the learned system significantly change?

Citations:

[1] Reali, Chater, Christiansen (2018), "Simpler grammar, larger vocabulary: How population size affects language", Proceedings of the Royal Society B

[2] Rezaii, Mahowald, Ryskin, Dickerson, Gibson (2022), "A syntax–lexicon trade-off in language production", PNAS

We thank the reviewer for engaging in an interesting discussion about our work. Before turning to our detailed response (below), we would like to highlight that none of the points that the reviewer raised seem to be grounds for rejection. We have already addressed the concerns that were raised in the review, and the reviewer’s main follow-up questions about the syntax-semantics interface are not specific to our work. These questions touch upon some of the deepest open challenges in cognitive science and linguistics. Our work does not attempt to address these challenges. Instead, we follow common practices in the field and make well-established simplifying assumptions in order to advance the understanding of how lexical systems may emerge from contextual pragmatic interactions. Furthermore, the reviewer’s first follow-up question, on estimating the expected human performance, has actually helped us to further strengthen the support for our results and conclusions. We therefore hope that the reviewer will positively reconsider the rating of our paper.

Detailed response to the reviewer's questions:

Firstly, could you quantify the expected human performance on the task and how you infer that number?

While the Mädebach et al. data does not allow us to quantify precisely the human performance in our task (our point in the paper, lines 226-233, and in our rebuttal was qualitative rather than quantitative), we can use that data to roughly estimate an upper and lower bound on the human performance. For the upper bound, notice that Figure 2 in Mädebach et al. shows that, across all context conditions (no-competitor, lexicon-sufficient, and syntax-necessary), the performance is around 80%, and recall that Mädebach et al. did not restrict the participants’ responses to lexical items. Therefore, it is unlikely that in our task, which is restricted to lexical items, the performance would be better than 80%. For the lower bound, notice that in the most favorable condition, i.e., the lexicon sufficient condition, which could in principle be solved using the lexical items, the proportion of responses that correspond to lexical items without syntactic construction is only ~42% (‘no specification’ and ‘lexical’ response type). In ~39% of the response participants use syntactic construction even though there is a lexical item that could unambiguously describe the target. One possible explanation for this behavior is that in some cases lexical retrieval is harder than syntactic constructions (e.g., for infrequent words). Therefore, while we expect the proportion of lexical responses to be higher than ~42% in our version of the task, we also expect the error rate to be higher. While we don’t have a way to quantify this further, we do agree with the reviewer that the task performance would probably be higher than 50%. To summarize: the Mädebach et al. data suggests that the human performance in our task would be somewhere between 50%-80%. In comparison, our model achieves ~72%, which is well within the reasonable range of human performance. We find this estimation very helpful to further support our results and conclusions, and thank the reviewer for suggesting it!

where do you draw the line between syntax and semantics in this particular setup?

Syntax corresponds to how single words can be combined into larger linguistic structures, such as phrases or sentences. Lexical semantics corresponds to word meanings. These two subfields of linguistics are largely separable, but indeed, as the reviewer notes, there are gray areas in between. However, these gray areas are irrelevant to our work because our agents simply cannot learn them. Recall that in our setup communication signals cannot be combined and in each communication act only a single signal can be generated. Therefore borderline cases like the noun compound, which require combining words, cannot emerge in our agents. Instead of lexicalized compounds, our agents would simply use a single communication signal, thus bypassing the issue that Mädebach et al. had to address in the human data.