Thank you very much for the insightful comments! We address your questions here:

1. Why only MMLU? We chose to focus on MMLU as it pulls questions from a wide set of knowledge domains (i.e. multiple subjects) and is somewhat challenging for models, while still making it straightforward to elicit epistemic markers and measure accuracy. Since MMLU includes a diverse variety of domains (e.g. math, history, ethics, science), we believe that our results are indeed broadly generalizable, at least across domains.
2. Google Translate API. See point 2 of our general response!
3. Inter-annotator reliability. See point 3 of our general response!
4. Classifier accuracy. See point 4 of our general response!
5. Learning effects. This should be mitigated by randomization: since the languages are shown in random order, and we take the average across all experiments, any difference we find should be due to the stimulus, not the ordering. (note that we also randomize questions within each language)
6. Does reliance generalize to monolinguals? This is a really good question! Due to limitations with data collection (Prolific is an English-language platform), we were only able to focus on bilingual speakers. We agree that future work should study whether this generalizes to monolingual speakers. Our intuition is that it would: the “listener model” employed by each bilingual speaker should be independent for each language, such that an English-French bilingual perceives French epistemic markers in the same way that a French monolingual perceives them. This should be empirically tested, though!
7. Omission of scores for few-shot models. Could you expand on what you mean here? The numbers in Table 3 (overconfidence and overreliance risk) are computed based on the generations of the few-shot prompted models.
8. Language support for Llama. This is a good point that we would have liked to spend more time on in the paper. In general: while we don’t observe differences in performance (indeed, Llama gets better MMLU accuracy in Japanese and Mandarin than German), we do notice that the distribution of generations for these languages is qualitatively different between Llama and GPT-4 (Figure 2). As we do not have access to the exact pre-/post-training recipe for Llama or GPT, it is hard to make statements about exactly why this difference exists. However, we hypothesize that it is due to the fact that GPT-4o is likely post-trained for Japanese and Mandarin, while Llama is likely not; Zhou et al. (2024) show that post-training can significantly alter the distribution of epistemic markers. This would explain why the distributions for English, German, and French are somewhat consistent across all models.

Thank you again for taking the time to write your review; we hope we've addressed your concerns.

Thank you for taking the time to write such a detailed review! We address your questions here:

1. Prompt in the appendix. The prompt in the appendix is an example of one of the English few-shot prompts we used. We sampled three few-shot prompts in total. Few-shot examples were translated across languages. We will release all prompts with the camera-ready.
2. Eliciting epistemic markers from few-shot prompts. See point 1 of our general response!
3. Plain language only in reliance study. Since we prompt models to produce epistemic markers in Section 3, models almost never fail to produce a marker. We include plain language in Section 4’s reliance study as a control. Plain expressions were hand generated.
4. If markers are rare without prompting, does this change our results? See point 1 of our general response! At a high level, no: overreliance risks are actually even higher for plain expressions than for generations including strengtheners.
5. Why is Chinese not included in reliance figures? See footnote 6 — because all models generally failed to generate weakeners in Mandarin, we excluded it from the reliance studies.
6. Plain language is most relied on across languages. We agree that this is an interesting result, and will add it to the main text for the camera ready!
7. Fleiss scores. See points 3 and 4 of our general response!

Again, thanks so much for the review, and let us know if this addressed your questions!

Thanks for your response!

I am curious about your "Why do we need few-shot prompting to elicit markers? Is this artificial?" response. If plain language is worse than markers and markers do not tend to be generated by models naturally (and if they are generated they are known to be somewhat over-relied on) doesn't that seem like we're investigating the wrong thing here?

There seems to be an argument here that suggests there is not much point in encouraging models to generate these type of markers. Or do you view it as the best of the available bad options?

Could you respond to the following? (Mainly the last part, that the markers within the prompts are not necessarily calibrated to the model being tested.)

I had one primary concern about the work's methods. There is something artificial about encouraging models to produce the strengtheners/weakeners via a few-shot prompt and then evaluating them. There seems to be more than one way for a model to interpret that type of prompting. (157-161). Note that the strengtheners/weakeners within the prompt themselves are not calibrated to the model's capabilities so that might introduce a confounding factor.

Thanks for the helpful feedback! We address your questions here:

1. Why elicit epistemic markers through few-shot prompts? See point 1 in our general response!
2. MMLU is US/Western-centric. We agree that MMLU is quite US/Western-centric! We acted to mitigate this in our experiments by removing culture specific questions. To find these, we took the intersection of MMLU and JMMLU. JMMLU replaces “Western culture” questions with equivalent “Japanese culture” questions. By taking the intersection of these two datasets, we removed (most) culture-subjective questions from the dataset. We will add text in the camera-ready to clarify this.
3. Google Translate API. See point 2 in our general response!
4. Underconfidence rates. While we don’t report these numerically in the paper, these are available in Figure 1: underconfidence is the probability of a correct response given a weakener, i.e. the leftmost bar in each facet. We will include a table with these numbers in the appendix of the camera ready! Note though that these are less concerning from a safety perspective: the risks of relying on an underconfident model are much lower than that of an overconfident one.

Thanks again for the insightful comments; hopefully this addresses your concerns!

Thanks so much for your detailed review and feedback! We address your questions below:

1. Definition of confidence categories. As we mention in the paper [Appendix A.1], confidence categories are assigned based on bucketing a 1-7 Likert scale (1-2 = weak, 3-5 = moderate, 6-7 = strong). We will make sure to clarify this in the camera ready.
2. Inter-annotator agreement across different languages. Could you expand on what you mean by this?
3. Cultural/linguistic effects on perception. We agree that cultural and linguistic factors influence the perception of confidence across languages! This is what we explore in Section 4 of the paper, and find that there is indeed cross-linguistic variation in reliance on epistemic markers.
4. Language selection. We chose the five languages in the paper (English, French, German, Japanese, Mandarin) as they are generally high resource in terms of language modeling, and because of the availability of fluent human participants. There are, of course, resource differences between them, and we do observe that this empirically affects performance: in English, models have higher MMLU accuracy (Table 1) and are generally better at instruction following (Table 5, though note French!). However, this does not impact our high level story, as MMLU accuracy does not vary significantly between languages (always between 75-80%).
5. Strategies to mitigate overconfidence. We believe that this is an exciting area for future work, but is beyond the scope of our paper. Our goal is to highlight (1) existing overconfidence risks and (2) issues with purely English-based approaches to studying overreliance. Prior work (e.g. Zhou et al. 2023, 2024) has explored strategies for mitigating overconfidence in English (e.g. cleaning pre-training data, re-constructing preference datasets), and future work should explore this in the multilingual setting!
6. Harmful manifestations of overconfidence. We will add some examples in the introduction of the camera ready! The canonical example from the literature is a situation in which an LLM is deployed in a high-risk setting (e.g. medical, legal), and stakeholders are told confident but false information.
7. Explicit examples of markers. We include these in Appendix A.3!
8. Explicit examples of prompts. We include an example in Appendix A.3!