Characterizing Context Influence and Hallucination in Summarization

Thank you for your constructive comments. We are glad that you found the idea of the paper interesting and impactful. We address your concerns below:

In this way, the authors should consider CID as a simple extension of CAD.

We agree that CID is a simple extension of CAD. We have revised Section 3.2 and the contributions paragraph in Section 1 to reflect this better.

Also, from the experimental results, the faithfulness score is not comparable when $\lambda=0.5$, which somehow shows that averaging prior and posterior logits is not a common practice for decoding with context although a smaller $\lambda=0.5$ can lead to a better privacy score.

Yes, our experimental results demonstrate a tradeoff between faithfulnes and privacy, where a better privacy score leads to a worse faithfulnes score. This tradeoff makes sense because the generations are relying less on the context ($\lambda=0.5$) and hence will be less faithful to the context. But the privacy will improve since less privacy of the context is likely to be leaked from the generations.

the prompts used in this paper only consider removing the whole context.

We actually did perform experiments that consider removing subsets of the context, which can be found in section 4.4. We accomplished this by removing n-grams of the context (contiguous subsets) and measuring the n-gram influence. Please look at the results in this section to help understand context influence. However, for the main results (Section 4.2), we only looked at context influence for an entire context. In our revision, we have added that the prompts used in Appendix C are only for the main results.

I would recommend the authors move some of the definitions and discussion in Appendix B to the main article for a clear presentation. Also, if the authors could accept my first major suggestion, I would recommend that the authors discuss more about privacy in the method section, while keeping CID as an extension of the preliminary.

We appreciate the reviewer's suggestions and accept them into our revision. In particular, we moved the DP definition to the preliminaries, added more discussion about DP in section 3.3, and removed some discussion of CID to keep it more as an extension of CAD.

What are the major technical contributions of CID compared to previous work?

Prior work has only focused on analyzing privacy leakage of pre-training data via memorization. Other works have looked into context attribution by analyzing contextual information, but is a different goal from analyzing privacy leakage. Some works have looked into privacy leakage of RAG, but they did not use context influence or DP and instead employed membership inference attacks. We are the first to define a measure for privacy leakage of contextual information via context influence and connect it with DP. We also introduced and proved a context influence-hallucination tradeoff. And finally, we are also the first work to look at how other factors such as model parameters, context size, response length, and n-grams of context affect privacy of context.

The authors seem to only evaluate removing all context in the experiment. I would be interested in seeing how removing a subset of context affects the output.

Please see section 4.4 which contains results on how removing a subset of the context affects the output.

Relating context influence with DP is novel, however, is there a way to distinguish the information? In other words, how do the authors determine which part of the information is private and which part benefits the generation quality?

This is a really good question that needs much more future work: Determining which subset of a context is private is a more realistic setup, as we could treat some parts of the context as private and the rest to be public information. However, we surmise at this time that this level of separation is difficult to achieve in an automated way without some sort of human intervention. And the question of which part benefits the generation quality is another line of work called context attribution, which is a different goal of ours but also relevant. Our n-gram influence results does give some insight into which parts benefit the generation quality. In particular, 128-gram subsets of PubMedQA documents have the highest influence on the model compared to other n-gram sizes.

Thank you for your constructive comments. We are glad that you found the idea of the paper interesting and impactful. We address your concerns below:

In particular, I suggest the authors change the presentation incorporating a figure with a running example of the problem.

We have included additional information relating Figure 1 to the main text with a running example in our revision.

The authors can just say it's a well-documented emergent behavior in LLMs.

We have revised the aforementioned sentence to reflect the reviewer's comment.

For instance, the words "context" and "hallucination" are consistently used without an appropriate definition

We define these terms in the first and second paragraph of the perliminaries. From our manuscript, we define $**context**$ "$D=[d_1,...,d_n]$ be some context document/text" and $**hallucination**$ "It is possible that the resulting response $\mathbf{y}$ contains fictitious information--- i.e., $\mathbf{y}$ is not supported by $D$--- which we deem as a hallucination by the LLM." Please let us know if there is any confusion here.

but I think the authors need to rethink how they present it, making it more self-contained.

We felt our description of CAD sufficied for understanding the methodology of our work, since it was carefully paraphrased from Shi et al., 2023. We would value your input in what we can cut/add to make the CAD discussion more precisely self-contained.

An important comment to make about Eq 2 is that it biases the distribution by making the model more confident when the document posterior diverges from the prior. I think this comment can compress most of the text there.

Entropy, which is a measure of uncertainty, can be still large even when applying Eq. 2. Actually, it can be the case that the model's prior output is quite confident, but its document posterior output is not confident. Van der Poel et al., 2022 included entropy in their formulation of Eq. 2 to account for this case. What Eq. 2 does is bias the LLM towards the posterior when it diverges from the prior without necessarily making the model more confident.

P-CXMI is only mentioned once in the paper, I still don't know what it is.

We presented P-XCMI in Eq. 1 of the preliminaries section. To clarify for the confusion, we have included a reference to Eq. 1 prior to introducing Def 3.1 in our revision.

do the authors mean that $D'$ is a substring of $D$? What's $f_{\text{Mem}}$?

Yes, $D'$ is a substring of $D$. And $f_{\text{Mem}}$ is a typo. We have revised these accordingly in our revision.

The author's main contribution is Eq 5. I'm not sure I see the novelty here other than simply down-weighing the context influence in the decoding.

Eq. 6 (and Thm 3.1) are the primary contributions of our work. Eq. 5 only reformulates the CAD framework from Shi et al., 2023, so we can directly appply that formulation in our context influence calculation. We have made this point more clear in Section 3.2 of our updated revision.

This statement is wrong. In general privacy is saying a divergence between the distribution coming from one dataset to the distribution coming from an adjacent dataset is small

We do not disagree with the broader definition of privacy you provided. But our statement merely follows prior work (Wang et al., 2015) which states that obtaining a sample from a distribution can inherently provide privacy (since sampling is not deterministic and instead stochastic). However, sampling alone does not bound the privacy leakage, and hence, the sample is not differentially private until we make more modifications to the CID framework (as is done in Appendix B).

What's the main question the authors are trying to answer? ... What's the takeaway with respect to the authors method?

The key takeaway from the main results is that relying too much on the context for mitigating hallucination leads to higher risk of privacy leakage. We theoretically demonstrated this with Theorem 3.1 and we experimentally showed this in section 4.2, where increasing the F1 ROUGE-L by 10% caused LLaMA to be influenced by the context 1.5x more. Table 2 qualatatively showed that this increase in context influence leads to regurgitation of the context, a direct privacy leakage. Additionally, we can make this connection in the experimental results between context influence and privacy because of section 3.3, which was not addressed in prior works. Moreover, the experimental analysis in Section 4.3 explored how context influence can be affected by important hyperparameters, such as model size, context size, and generation length. We discuss the privacy implications of these results in Section 6.

Thank you for your constructive comments. We are glad that you found the idea of the paper interesting and impactful. We address your concerns below:

there is no evaluation of the method whatsoever nor a relevant privacy metric in an adversarial learning setting (e.g., private attribute inference attacker [1]) that quantifies how well the proposed method is able to protect users from such attacks.

Thanks for sharing this line of work. Our work is motivated by Differentially Privacy (DP)-centric privacy leakage gurantees which provide a bound on the amount of information leaked, without any assumptions made by the attacker. In Section 3.3, we showed that our context influence definition is intimately connected with DP. More specifically, conext influence exactly follows the definition of DP except that we have fixed the private dataset $D$, the neighboring dataset $D'$, and the output $y_t$. Context influence measures the output change if one were to remove a substring from the context. Large output change means the model relied heavily on the substring, which is a privacy risk. Hence context influence gives a strong lower-bound on the $\epsilon$ value used to measure privacy leakage with DP. Given this, we believe that Context Influence robustly measures the privacy leakage of a provided context/document when sampling from an LLM, compared to privacy attribute inference [1] which instantiates an attacker by using adversary learning to measure a particular privacy leakage using attribute inference with AUC. Morevoer, it is not entirely clear how we can use the same attacker in [1] to our setting, but it can be an interesting follow up work.

In fact, except for the connection between the proposed method and DP mentioned in the appendix, I don’t see any other relevant information in the paper

We have added more discussion about context influence's connection with differential privacy in Section 3.1 and 3.3 of our revision, as well moved relevant privacy definitions to the preliminaries, which hopefully makes the privacy part of our paper more clear.