Uncovering Gaps in How Humans and LLMs Interpret Subjective Language

Thank you for your review! We’re glad you thought our high-level motivation is important and intriguing, and appreciate your feedback. Based on the feedback we’ve done the following:

• Added the experiment you requested where we compare the cosine similarity of the gradients for the same phrase over different prompts; we find that the cosine similarities are routinely high.
• Added descriptions of how the failures TED uncovers could be resolved with standard methods
• Added more detailed descriptions to Figure 1, along with additional writing changes.

We’ve also responded to your specific comments below. We hope that if this helps assuage your concerns, you’ll consider raising your score.

In line 207, "we average over gradients from n generic prompts", I am interested in the consistency of the operational semantic vectors of the same word w. Could you present the inner-consistency of the vectors (e.g. distribution of their cosine similarity) of the same word w under n different prompts? If the distribution is very divergent, I would highly doubt the effectiveness of the calculated operational semantics.

We’ve included a figure with the cosine similarities for different terms in Appendix B.8 (see Figure 5); as you suggest for different phrases, we compute the gradients for many prompts and compute histograms of the pairwise similarities. Overall, we find that the cosine similarities are almost always very high. We also expect there to be some noise in individual examples (since specific outputs to each prompt are stochastic), which is why we average over many gradients to compute embeddings.

However, our empirical results are the main justification of the effectiveness of the calculated operational semantics; as we describe, the only difference between TED and the semantic only baseline is filtering based on our operational semantics. If the operational semantics were not effective, TED would not outperform the semantic only baseline, while in practice it does by large margins

The authors also did not discuss how TED can potentially benefit LLMs research and applications in this paper (correct me if I missed). To enhance its practicality, I believe it is good to conduct research and experiment to showcase how to improve LLMs' generation using the proposed TED. However, this paper does not include this part, which diminishes its significance.

Thanks for this suggestion; in the original draft, we discuss one potential method for improving models (training the LLM so its operational thesaurus matches the reference), in lines 525-531. Based on your suggestion, we’ve also added more actionable guidelines on how TED might improve systems; these include helping practitioners choose better system prompts, curating fine-tuning sets based on them, or using the human thesaurus as a direct supervision signal (see response to all reviewers). We think both adjusting the prompt and fine-tuning could be implemented easily with standard methods; we view the primary challenge and contribution of our work as finding these failures out of the large space of possibilities.

The authors should expand Figrue 1, add more descriptive text on it, and relate to the corresponding sections. Also, the authors is recommended to highlight that Step 2 produces a set of pairs that LLMs misunderstand, which is then evaluated in Step 3

Thanks for pointing this out; we’ve adjusted this figure in the following ways:

• We change orthogonal and perpendicular to similar (SIM) and dissimilar (DIS)
• We changed the title of Step 2 to “Identify Failures (Clashes)” to make it clear these are things the LLMs misunderstands
• We include a more clear arrow from the pair in Step 2 to the output in Step 3 to make it more clear that the failures come from Step 2. We additionally made the label of Step 3 "Test Failures downstream" so it more clearly links from Step 2.
• We made the caption more descriptive, and added examples

We hope this makes it easier to parse, and helps resolve your concerns.

I think the prominent problem is the writing, which fails to convey the authors' idea clearly. I've read the paper word by word multiple times, but untechnical issues making the paper logically unsmooth and difficult to understand, such as missing clear description of key parts.

We respond to many of the specific writing points you raise below, but one high-level design choice that we think caused confusion is our decomposition of Section 3 and Section 4; in Section 3, we describe TED and ways to evaluating TED in the abstract, then include our specific instantiation (with many of the details you asked for in Section 3) in Section 4. We did this because TED is general, and we expect it to work for many choices of prompt, embedding, and judge. In the revision, we forward-reference that we decompose in this way at the beginning of Section 3 to make this more explicit.

[response continued below]

Thank you for your review of our work! We’re glad you found it offers a “fresh, nuanced perspective to alignment research”, that we “address a crucial challenge”, and our “methodology is rigorous”. We respond to your questions and concerns below.

The paper relies on GPT-4 for validating downstream failures, but this could introduce bias since GPT-4 is itself an LLM with its own alignment characteristics. Exploring alternative or supplementary validation approaches, such as using human expert reviews or a consensus-based scoring system from multiple models, could mitigate this potential bias. Given the reliance on GPT-4 to validate TED’s flagged misalignments, how do you address potential alignment biases that GPT-4 might introduce?

Thanks for raising this; we agree, and we’ve added human validation on one part of the pipeline that doesn’t currently have human supervision (supervising GPT-4 on the judge over outputs). We find that humans tend to agree with LLMs, and that if anything LLMs underestimate the rate at which TED produces failures; see details in the response to all reviewers and Appendix B.7.3 of the revised paper.

The paper mentions embedding each phrase independently of context, which may overlook nuances in interpretation that depend on the type of task (e.g., writing a "witty blog" versus a "witty proposal"). Incorporating context-dependent embeddings could enhance TED’s robustness by tailoring thesaurus creation based on usage scenarios. This could be achieved by developing context-specific operational thesauruses or dynamically updating embeddings based on task context, potentially using attention-based methods to focus on relevant contextual tokens.

This is a good point; we discuss this in lines 474-481 of our submission. One benefit of TED is that it puts no constraints on what counts as a phrase, so it can seamlessly extend to these settings.

How does TED handle potential context-specific interpretations of subjective phrases (e.g., “witty” in different tasks like blogs vs. proposals)?

Right now TED does not handle context, although TED could be seamlessly extended to do so (by only averaging gradients where a term is used in a specific context). We think this is an exciting direction for subsequent work.

Did you observe specific hierarchies or dependencies between phrases (e.g., “intelligent” often implying “engaging”)? If so, how did TED handle these dependencies?

Right now TED doesn’t handles these dependencies; it’s currently symmetric. We expect that there are ways to extend TED to handle them down the line; for example, we could measure the gradient of “engaging” outputs relative to “intelligent” prompts and vice versa; if “intelligent” implies “engaging” but not the reverse, the gradients should be large on “engaging” outputs but not on “intelligent” outputs. We think adding this as a post-hoc step is an exciting direction for subsequent work.

Beyond identifying misalignments, have you considered how TED’s findings might guide improvements in LLM training or alignment processes?

Great question; in the original draft, we discuss one potential method for improving models (training the LLM so its operational thesaurus matches the reference), in lines 525-531. We’ve also added more actionable guidelines on how TED might improve systems; these include helping practitioners choose better system prompts, curating fine-tuning sets based on the , or supervising on the internal thesaurus directly (see response to all reviewers). We think both adjusting the prompt and fine-tuning could be implemented easily with standard methods; we view the primary challenge and contribution of our work as finding these failures out of the large space of possibilities.

Please let us know if you have any additional questions!

Thanks for your review of our work! We’re glad you thought our approach is “interesting” and is “filling a critical gap in aligning LLM behavior with human intent”. We respond to your questions and comments below.

While TED detects misalignments effectively, the paper could provide more actionable guidelines or solutions to mitigate these issues in real-world applications.

Thanks for this suggestion; in response we’ve also added more actionable guidelines on how TED might improve systems; these include helping practitioners choose better system prompts, curating fine-tuning sets based on them, or using the human thesaurus as a direct supervision signal (see response to all reviewers). We think both adjusting the prompt and fine-tuning could be implemented easily with standard methods; we view the primary challenge and contribution of our work as finding these failures out of the large space of possible behaviors.

The evaluation metrics for TED’s effectiveness could be clarified, as the success rates reported could benefit from further contextualization to understand their practical implications. Also, it would be better to clarify how exactly the success rate is calculated.

We describe the calculation of the success rate in lines 231-234 of the original paper, and the actual implementation details we use in the Evaluating TED paragraph of section 4.2. Intuitively, the success rate for a pair of phrases describe what fraction of the time (on held out prompts) including the phrase produces the unintended consequence or the inadequate updates; this aims to capture how frequently the failure arises at deployment. Both high and low success rates have important practical ramifications; a low success rate (e.g., 0.1) could still be too high for failures like producing “dishonest” outputs when humans prompt for enthusiasm. The aggregate success rates, described between lines 235 - 240 and reported in the Figures, primarily capture TED’s overall efficacy.

However, the implications of these results need clearer justification and relevance to real-world LLM applications. For instance, the statement "TED additionally finds inadequate updates with higher success rates than the semantic-only baseline, but both TED and the baseline find fewer failures overall" is an interesting finding, but it’s unclear what this means in practical terms.

The main observation in this statement is that inadequate updates seem to occur less frequently (or are at least harder to find for both the semantic only baseline and TED) than unexpected updates; in the original version, we say “[this] indicates that inadequate updates are less frequent in practice than unexpected side effects.” We might intuitively expect this since there are more ways of adding unexpected attributes than missing necessary attributes. This also practically means that developers should actively test for a wide range of unexpected side-effects.

In line 283, "For example, the LLM might language model to write a “witty” essay or an “accessible” blogpost about machine learning." What does "the LLM might language model" mean?

Thanks for flagging this; we mean a “user might prompt an LLM to write…”. We updated this in the revision.

Please let us know if you have any additional questions!

Thank you for your thoughtful review of our work! We’re glad you thought our method is “interesting and creative”, and that our contribution of “characterizing LLM behavior via relationships” to be “very compelling”. We respond to your comments below:

The authors rely on GPT-4 in many parts of the pipeline: identifying subjective phrases, constructing the reference thesaurus, and acting as a judge. These decisions are well-motivated but some human validation is needed at each stage, perhaps on just a small sub-sample. Especially because this work focuses on biases and limitations of LLMs, it’s hard for me to switch to fully trusting GPT-4’s outputs.

Thanks for raising this; we agree, and we’ve added human validation on one part of the pipeline that doesn’t currently have human supervision (supervising GPT-4 on the judge over outputs). We find that humans tend to agree with LLMs, and that if anything LLMs underestimate the rate at which TED produces failures; see details in the response to all reviewers and Appendix B.7.3 of the revised paper.

For the human evaluation, there should be some sort of inter-annotator agreement recorded (again, maybe on a small subset). Because we’re dealing with subjectivity, humans’ interpretations of each phrases and the relationship between phrases would likely vary widely.

This is a good point; in our study we have three annotators label each pair of phrases, and only include pairs where all three annotators agree (see 315-316). We also report aggregate statistics about agreement in Figure 4; the annotators all agreed on around 60% of pairs (whereas they only would have agreed on 11% via random chance).

Given that this work is ultimately about meaning and the word semantics is used throughout the paper, there is surprisingly no connection to any relevant concepts or prior literature in subjectivity, semantics, or pragmatics. [...] I’d recommend connecting this work to ideas in semantics and pragmatics that focus on the relationships between statements: entailment/natural language inference, presuppositions, and implicature. I think this could help ground the method, annotation, and evaluation.

Thanks for raising this! We definitely think there are close relationships to concepts from the prior literature, and added a paragraph in related work to describe this. We think our work closely ties with pragmatic implication; it’s possible that some of the failures come from LLMs imitating humans’ pragmatic implications of phrases (e.g., it’s possible that humans actually tend to say more hateful things when asked to be witty.)

That being said, TED’s only goal is to uncover scenarios when using subjective phrases in the prompt will produce outputs that are different from what prompters expect, without attributing cause. Our evaluation directly tests for this: we ask human annotators (that simulate prompters) whether or not they expect outputs to have certain properties, and view any deviation from these expectations as failures. These failures will become especially salient as models are prompted by humans, but deployed with limited oversight.

One area that remains unclear to me is in the motivation for testing TED on LLM responses to this particular kind of ethical question. The examples in the prompt all start with “why is it okay”, which implies that “it is okay” which may bias both LLM outputs and the constructed thesaurus.

This is a great question; the ethical questions come up in the output editing setting that is designed to mimic Constitutional AI. In this setting, the goal is to get the model to adhere to a constitution, especially when responding to challenging topics with potential bias. However, TED finds failures in the inference steering setting too, where we use much more standard prompts (e.g., “Write an enthusiastic letter to a school board proposing the introduction of coding classes in middle schools”), so this is not a critical part of the setup.

More specifically, I'm not convinced based on the provided examples that unexpected side effects necessarily indicate model failures, as suggested by the language throughout the paper. I can't easily imagine what it would mean for subjective phrases to affect LLM outputs along just that one dimension (enthusiastic, witty, etc.) without affecting other dimensions?

To clarify, we say that a failure occurs when outputs are affected along directions that human annotators don’t expect, rather than all directions; this means that most effects are not considered failures (e.g., prompting for “enthusiastic” and getting “energetic” or “long” would not be failures). When collecting e.g., unexpected updates, we allow the annotator to specify whether or not they expect that prompting a model with one phrase affects the output along another axis, and only count failures that go against all annotators expectations.