Attributing Culture-Conditioned Generations to Pretraining Corpora

Thank you for your insightful review and very helpful comments! Below are our responses and we have updated the draft accordingly.

W1: While the methodology appears sound, my only concern is the limited scope of the study, which focuses solely on the OLMo-7B model. As a result, the findings might be seen as a case study specific to this model. It would strengthen the paper if the authors included analyses for at least one additional LLM to broaden the generalizability of their findings.

We quickly collected culture-conditioned generations for both food and clothing on olmo-7b-0424 (OLMo 1.7) which is trained on Dolma 1.7. Although this is the same model family as OLMo-7B, Dolma 1.7 contains pretraining documents not in Dolma 1.5, and OLMo 1.7 is trained with an updated algorithm from OLMo 7B. Other models supported by the WIMBD API we used, such as Pythia, are not particularly capable of instruction following culture-conditioned generations, and therefore, analyzing their generations is less informative.

Due to the time constraints of the rebuttal, we are unable to completely reproduce all analysis done in the paper. We analyzed 2 correlations (added to Appendix E.2):

1. Between the number of cultures independent symbol was generated for and its pre-training count: | Model | Category | Spearman Correlation | Kendall Correlation | |-------------|-----------|----------------------|---------------------| | OLMo 1.7 | Food | 0.416 | 0.313 | | OLMo 1.7 | Clothing | 0.507 | 0.362 | | OLMo 7B | Food | 0.358 | 0.260 | | OLMo 7B | Clothing | 0.521 | 0.367 |

We see that even though models and training data are different, the Spearman and Kendall correlations for food and clothing remain the same (both moderate-to-strong correlations). This means that the number of cultures an independent symbol was generated for and its pretraining count is positively correlated, regardless of the model.

2. Between number of memorized symbols and pre-training counts (OLMo 1.7 is sampled on 10 cultures due to time constraint, detail see Appendix E.2):

For this experiment, exhaustively searching for all memorized symbols of all 110 cultures requires running MEMOed on all symbol-culture pairs, which is not feasible due to the rebuttal time constraint. Nonetheless, we see that for clothing and food, the correlation of Olmo 1.7 increases despite the small number of cultures we evaluated. Therefore, the conclusion still holds that higher pretraining counts of cultures increase the number of memorized symbols in culture-conditioned generations.

W4. While this study highlights the existing problems with underrepresented cultures in pretraining corpora from a new perspective, it fails to address or propose potential directions for solving these issues. Without this, the paper remains another verification of known problems, which is still valuable but not particularly groundbreaking. The authors should discuss how their findings could inform improved pretraining data/methods or mitigation strategies that do not require changes in pretraining.

We politely disagree that our paper is “another verification of known problems.” While our paper did highlight the problems with underrepresented cultures in pretraining corpora, we also introduced a framework for identifying cultures with insufficient memorized symbols and symbols with overmemorization from high frequency. These contributions are novel compared to concurrent works on evaluating culture biases in LLM generations and can be integrated with existing bias mitigation or unlearning strategies to solve culture biases.

Many works have been done on unlearning knowledge, mitigating biases, or combatting spurious patterns through influential data [1][2][3][4][5]. These works follow the pipeline of influential pattern attribution, mostly through influence function [6] or feature attribution [7], and then demote the patterns by either learning confound-invariant representations, e.g., adversarial training [8] or data-based approaches to balance the training data, e.g., counterfactual data augmentation[9]. Our discovery of symbol overmemorization can be an alternative to the first step, as it helps identify high-frequency symbols that are biasing model generations, which are the targets of demotions in the second step. Our discovery of cultures with inadequate memorized symbols can also facilitate strategies for data augmentation: from pretraining documents containing these cultures, we find symbols that are related to the culture but not generated by the models, and these will be the symbols that need augmentation.

[1] Mitigating Social Biases in Language Models through Unlearning

[2] Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updates

[3] Adversarial Unlearning: Reducing Confidence Along Adversarial Directions

[4] Debiasing Machine Unlearning with Counterfactual Examples

[5] Unlearn What You Want to Forget: Efficient Unlearning for LLMs

[6] Understanding black-box predictions via influence functions

[7] “Why should I trust you?”: Explaining the predictions of any classifier

[8] Adversarial removal of demographic attributes from text data

[9] Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology

Q1. In Figure 1, the top-down order does not match the numerical order. Why are memorized symbols shown at the bottom?

In Figure 1, we order the symbol categorizes based on their level of “grounding” on pretraining documents. Memorized symbols are most grounded to pretraining documents due to the fact that these symbols are generated for the culture because of high symbol-culture co-occurrence in pretraining data; generalized symbols are traced to memorized symbols as evidenced from our analysis in Section 4.4; independent symbols do not have any evidences from pretraining document that associate them with cultures, and therefore it is on the top of the pyramid.

Q2. In Figure 3, what does "overgeneralization" refer to? It is not mentioned in the text. Do you mean "overmemorization" instead? The same applies to the caption of Table 3.

Thank you for catching this typo. We have changed all "overgeneralization" to "overmemorization".

Q3. How are culture-referring n-grams defined for the Document-Signal to Noise Ratio?

In L220, we define culture-referring n-grams as the n-grams associated with that culture’s country name and its corresponding nationality for identifying people from that culture. For example: India and Indian; China and Chinese; South Africa and South African, etc.

Q4. Why do the memorization classification criteria differ for cases where n(C_G) > 5 and n(C_G) < 5?

The memorization classification criteria differ for cases where (n(C_G) > 5) and (n(C_G) < 5) because, based on empirical observations, the Z-score becomes a misleading metric when the sample size ((n(C_G))) is small (i.e., <=5). This is because distributions with such small sample sizes often do not approximate a normal distribution well, meaning extreme Z-scores, like 2.6, are unlikely to be valid or meaningful. Therefore, we apply different criteria for small and large (n(C_G)) to account for the differences in distribution behavior and statistical reliability.

Thank you for your insightful review and very helpful comments! Below are our responses and we have updated the draft accordingly.

W1. The study is limited to only one pretraining corpus and one LLM, which is understandable given the scarcity of open resources.

We quickly collected culture-conditioned generations for both food and clothing on olmo-7b-0424 (OLMo 1.7) which is trained on Dolma 1.7. Although this is the same model family as OLMo-7B, Dolma 1.7 contains pretraining documents not in Dolma 1.5, and OLMo 1.7 is trained with an updated algorithm from OLMo 7B. Other models supported by the WIMBD API we used, such as Pythia, are not particularly capable of instruction following culture-conditioned generations, and therefore, analyzing their generations is less informative.

Due to the time constraints of the rebuttal, we are unable to completely reproduce all analysis done in the paper. We analyzed 2 correlations (added to Appendix E.2):

1. Between the number of cultures independent symbol was generated for and its pre-training count: | Model | Category | Spearman Correlation | Kendall Correlation | |-------------|-----------|----------------------|---------------------| | OLMo 1.7 | Food | 0.416 | 0.313 | | OLMo 1.7 | Clothing | 0.507 | 0.362 | | OLMo 7B | Food | 0.358 | 0.260 | | OLMo 7B | Clothing | 0.521 | 0.367 |

We see that even though models and training data are different, the Spearman and Kendall correlations for food and clothing remain the same (both moderate-to-strong correlations). This means that the number of cultures an independent symbol was generated for and its pretraining count is positively correlated, regardless of the model.

2. Between number of memorized symbols and pre-training counts (OLMo 1.7 is sampled on 10 cultures due to time constraint, detail see Appendix E.2):

For this experiment, exhaustively searching for all memorized symbols of all 110 cultures requires running MEMOed on all symbol-culture pairs, which is not feasible due to the rebuttal time constraint. Nonetheless, we see that for clothing and food, the correlation of Olmo 1.7 increases despite the small number of cultures we evaluated. Therefore, the conclusion still holds that higher pretraining counts of cultures increase the number of memorized symbols in culture-conditioned generations.

W2. This study relies on searching for symbols in culture-conditioned generations within the pretraining data and provides a relational analysis. However, it does not guarantee that the selected training documents are causally decisive for the symbols in question. Incorporating influence functions could provide insights into causal relationships. While the computational cost might be an issue, they could be applied to a subset of the dataset or specific experiments.

Thank you for your suggestion. Currently we are running experiments on influence functions on pretraining documents and will update the results in a few days.

W3. Lines 427-430 require further explanation. What do these correlations imply?

Thank you for pointing this out. We found an error in the calculation, and here are the updated correlations:

• Clothing: Spearman Correlation: 0.551; Kendall Correlation: 0.385
• Food: Spearman Correlation: 0.519; Kendall Correlation: 0.380

We see a moderate to strong positive correlation between 1) the average over ratios of the document count of an independent symbol to the document count of every memorized symbol and 2) the number of cultures that the independent symbol is generated for. This suggests that the higher frequency the independent symbol appears in pretraining data, the more cultures the symbol is generated for. This result shows the reason why an independent symbol, which is not associated with any culture in pretraining data, is generated for that culture is due to the symbol itself having a very high frequency in pretraining data.

Thank you for your positive review and very helpful comments! Below are our responses and we have updated the draft accordingly.

W1: Reliance on a Single Model: The analysis focuses solely on the OLMo-7B model and its pretraining dataset, Dolma. It is unclear from the analysis how the conclusions would vary on other models or models of other sizes.

We quickly collected culture-conditioned generations for both food and clothing on olmo-7b-0424 (OLMo 1.7) which is trained on Dolma 1.7. Although this is the same model family as OLMo-7B, Dolma 1.7 contains pretraining documents not in Dolma 1.5, and OLMo 1.7 is trained with an updated algorithm from OLMo 7B. Other models supported by the WIMBD API we used, such as Pythia, are not particularly capable of instruction following culture-conditioned generations, and therefore, analyzing their generations is less informative.

Due to the time constraints of the rebuttal, we are unable to completely reproduce all analysis done in the paper. We analyzed 2 correlations (added to Appendix E.2):

1. Between the number of cultures independent symbol was generated for and its pre-training count: | Model | Category | Spearman Correlation | Kendall Correlation | |-------------|-----------|----------------------|---------------------| | OLMo 1.7 | Food | 0.416 | 0.313 | | OLMo 1.7 | Clothing | 0.507 | 0.362 | | OLMo 7B | Food | 0.358 | 0.260 | | OLMo 7B | Clothing | 0.521 | 0.367 |

We see that even though models and training data are different, the Spearman and Kendall correlations for food and clothing remain the same (both moderate-to-strong correlations). This means that the number of cultures an independent symbol was generated for and its pretraining count is positively correlated, regardless of the model.

2. Between number of memorized symbols and pre-training counts (OLMo 1.7 is sampled on 10 cultures due to time constraint, detail see Appendix E.2):

For this experiment, exhaustively searching for all memorized symbols of all 110 cultures requires running MEMOed on all symbol-culture pairs, which is not feasible due to the rebuttal time constraint. Nonetheless, we see that for clothing and food, the correlation of Olmo 1.7 increases despite the small number of cultures we evaluated. Therefore, the conclusion still holds that higher pretraining counts of cultures increase the number of memorized symbols in culture-conditioned generations.

W2: It is not clear to me how the definitions of what constitutes memorization (e.g. training document classification) might change the analysis?

In this work, we study memorization from the perspective of cultural knowledge, and we wish to determine whether a symbol S is generated for culture C because the co-occurrence between culture C and symbol S is memorized by the model from the pretraining data. Many works on correlating model performance with knowledge memorization use the co-occurrence of input-output n-gram pairs in pretraining corpora to approximate memorization [1][2][3], and our work on using input-output relevant training documents as definition for memorization falls in this line of work.

Other definitions of memorization, such as exact matches of generation [4] or influential examples discovered by influence functions [5], may only impact the categorization of which symbols are memorized. The rest of the analysis, including how memorization is correlated with pretraining frequency of cultures, as well as studies on symbol overmemorization, culture overmemorization and traceable generalization is not impacted by the definition of memorization. To examine how different definitions of memorization will impact the conclusions on memorization capabilities of models on cultural knowledge would be future work.

[1] Generalization vs. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data

[2] Large Language Models Struggle to Learn Long-Tail Knowledge (ICML 2023)

[3] When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (ACL 2023)

[4] Quantifying memorization across neural language models (ICLR 2023)

[5] Studying Large Language Model Generalization with Influence Functions (Anthropic)

Thanks for the detailed response and updated experiments.

Thank you for your positive review and very helpful comments! Below are our responses and we have updated the draft accordingly.

W1: Presentation of relatively shallow experiments, limited technical novelty, and limited scale of experiments.

We politely disagree.

We provided in-depth experiments that explored four reasons why a symbol is generated by an LLM in culture-conditioned generations, finding significant correlations of model behavior with pretraining data, such as:

1. positive correlation between the number of memorized symbols for a culture and the count of documents in which the culture appears in the pretraining corpora, suggesting that it is more difficult for models to memorize symbols for cultures appearing in less frequency in pretraining data. This finding urges future work to augment the pretraining data for long-tail cultures or improve the pretraining algorithm to improve memorization in low-frequency settings.

2. positive correlation between the average ratio of an independent symbol to all memorized symbols and the number of cultures that the independent symbol is generated for, suggesting that the reason why a symbol that is not associated with any culture can nonetheless be generated for many cultures is that the symbol itself has a very high frequency in pretraining data. This finding urges further research to increase model compliance to prompts against high-frequency but less desirable generations.

3. positive correlation between how often a culture’s memorized symbol is generated for some other cultures and the number of appearances in topic-related pre-training documents, suggesting that the model is more likely to generate memorized symbols of cultures with higher frequency, despite the generation not being conditioned on those cultures. This finding suggests that models are able to generalize to new relations from pretraining data given related concepts.

Our paper’s technical novelty includes 1) developing MEMOed, a new systematic approach for determining whether symbols generated by an LLM are due to memorized co-occurrences in pretraining data, overmemorization from high-frequency entities or cultures, and generalization that can be traced to concepts that are related to the symbol. As the reviewer also mentioned, our approach can be generalized to other memorization scenarios where facts/information are generated conditioned on context. 2) introducing Document-SNR combined with token and sentence distance as a novel metric for determining document relevance to a pair of concepts, which, by our knowledge, is the first work to introduce an SNR-like metric to pretraining data analysis.

Our paper’s scale of experiments is by no means small: we studied 110 cultures, where model generations contain 2370 unique symbols for food and 1002 for clothing. In total, we have found 19.4 million documents to be relevant to the memorization of symbol-culture pairs in food and 7.85 million documents to be relevant to the memorization of symbol-culture pairs in clothing.

Dear Reviewer gSz2,

Thank you again for your review! We are happy to hear that you appreciated the usefulness of our research problem and generalization of our approach.

Based on your thoughtful feedback, we wrote a detailed rebuttal covering the following points:

• Justification on our technical contribution, scale of experiments and depth of analysis
• Clarification on our definition of memorization and how it is different from measuring generation consistency

We would love to hear your thoughts about our rebuttal, including whether it sufficiently addresses your concerns and questions. If you believe that our rebuttal is satisfactory, it would be great if you could consider increasing your score. Any feedback is welcome and greatly appreciated!

Sincerely,

Paper13342 Authors

To expand on my complaint about the overuse of mathematical notation, here's an example: (I excerpted this out from my first rebuttal response to make it easier to follow)

Introducing equations for the metrics is ok, but going so far as $r$ for relevance (which only ever takes a binary value) using the formulas in l255-257, and $m$ in l296-299 is just extremely reader unfriendly. For me reading this was basically:

ok, $Q$ is a pair of $[C,S]$, ok so $r$ is .... ok $r$ means relevant so $r(D,Q)$ is what? A culture or symbol? Oh right $Q$ is $[C,S]$ so $r(D,Q)$ means that document $D$ is $r$elevant to culture $C$ and $S$ in $Q$...

You could easily (and more clearly!) convey document relevance by just saying something like

We consider a training document to be relevant in determining if a symbol $S$ is memorized for culture $C$ if the either the token or sentence distance are sufficiently low along with document SNR requirements, or [l260-265]

Using this weird notation of setting a binary value to variable $r$ that never gets reused anywhere to 1 instead of just saying "under this condition a document is relevant" is very reader unfriendly and is emblematic of presentation issues throughout the paper.

This writing style basically makes p5 and 6 (explaining the method) pretty burdensome to follow, and frankly feels forced to impress the modal ICLR reviewer.

Thank you for the detailed response! Putting "limited novelty" in the review was a lazy and out-of-line comment on my part, so I would understand even an impolite disagreement!

"Shallow experiments" was really meant to convey my dissatisfaction with the level of detail provided for the experiments in the main text. Instead, a lot of space is wasted on kind of superfluous math, which is a common trap for ICLR/NeurIPS papers imo.

1.5 pages are devoted to an overly complicated explanation of basically "we use thresholds of our three scores to select documents that count toward memorization classification, and then we use the ratio of counted documents for the [symbol, culture] over all documents containing the symbol to determine if the symbol is memorized for a culture" using lots of unskimmable equations with unclearly defined (mathematical) symbols (for ex, $r(D,Q)$ is only defined as "considered relevant... i.e.$r(D,Q)=1$". You never say what $r$ even is!

Despite those 1.5 pages, not a single line in the entire main text clearly states the process by which the generated culture/symbol pairs are elicited or annotated! (though I appreciate you adding this to the appendices) Generally, it is good to tell the readers what the experiments are, clearly before you start giving them results. Instead I have to go back and forth, find the number of cultures tested then go back up, find the number of symbols tested, number of generations, etc. **You need to open the results with a clear explanation of what the experiments are; something like "using the following prompts we generated X total sentences for Y concepts in Z cultures and then [annotation protocol]." Even now after going through this paper a few times, I just now understand how even the symbols are produced: you use the final tokens outputted from two template sentences (uncovered via detective work and the new prompts appendix section). In the main text, this should be very clearly highlighted!

Overall, I am reluctant to recommend this paper for publication due to these presentation issues.

That being said, I would like to acknowledge the technical novelty. (but only using it on two prompt templates for the main cultural symbol generation process is a little to limited imo) I think this is an exciting approach and hope to see it published, but please work on writing a more accessible paper that doesn't require so much backtracking to understand, and clearly explain the methodology for your empirical results in the main text.

I have raised my soundness score, but will keep the others. I have updated my review text.

(Sorry to acknowledge the rebuttal so late!)

Thank you for your helpful comments and insightful questions! We have grouped your feedback and questions into some common themes. Please let us know if any of your questions remain unaddressed. Based on your feedback, we have updated the draft and provided the responses below.

W1: Justification for Equation (1).

First, we would like to clarify that Equation (1) is not our metric to determine memorization; rather, this equation defines one metric that we use to classify the relevance of one pretraining document D to a culture C. We name this metric “Document-Signal to Noise Ratio” because it is inspired by the Signal-to-noise ratio (SNR or S/N) measure that is used in science and engineering that compares the level of a desired signal (meaningful input) to the level of background noise (meaningless or undesired input) [1]. SNR is defined as the ratio of signal power to noise power, and a ratio higher than 1:1 indicates more signal than noise.

In our work, we categorize the desired signal (meaningful input) as the count of culture C in the pretraining document and background noise (undesired input) as the sum of counts of all other cultures (plus a small epsilon). We compute base-2 log over this ratio mainly for better interpretability as a d_SNR >= 0 means the count of culture C is at least as many as the sum of counts of all other cultures in this document, suggesting that this document is highly relevant to the culture.

[1] https://en.wikipedia.org/wiki/Signal-to-noise_ratio

W2: Justification for r(D,Q) in Criterion for Training Document Classification

r(D, Q) = 1 means that our MEMOed method classifies a pretraining document D as relevant to both symbol S and culture C, given that symbol S appears in the generation that is conditioned on culture C. There are two cases where we r(D, Q) = 1.

In the first case, d_SNR >= 0 means the count of culture C is at least as many as the sum of counts of all other cultures in this document, suggesting that this document is highly relevant to the culture, and d_TOK <= 2048 means the subtoken distance between culture C and symbol S is shorter than the context length of the language model (here we are using olmo-7b, but this can be adjusted for other models accordingly), and therefore enables the culture C and symbol S to appear in the same example during pretraining and ensure that their co-occurrence gets learned by the model.

In the second case, d_SENT <=2 means that the culture C and symbol S are occurring in adjacent sentences, suggesting that the local context is highly likely to discuss the relationship between culture C and symbol S and that this context is helpful for the model's learning of the relationship. We added an additional condition of d_SNR between -1 and 0 so that the global context (pretraining document) is also somewhat about culture C and not entirely irrelevant.

W3: Justification of measure for memorization.

In this work, we study memorization from the perspective of cultural knowledge, and we wish to determine whether a symbol S is generated for culture C because the co-occurrence between culture C and symbol S is memorized by the model from the pretraining data. Many works on correlating model performance with knowledge memorization use the co-occurrence of input-output n-gram pairs in pretraining corpora to approximate memorization [1][2][3].

We build on top of this co-occurrence perspective to define memorization. Contributory Score calculates the number of documents that contain the symbol S and that contribute to the memorization of the relationship between the symbol S and culture C. We then form a distribution of the contributory score of all cultures for which the symbol S is generated. The intuition behind this is that if the distribution of the contributory scores of all cultures is flat, then the symbol is not distinguishably associated with any of the cultures; only if the distribution of the contributory scores is spiked at a few cultures, then the symbol is associated with those cultures. Therefore, we find cultures that are “outliers” within the distribution using z-score=2.6 as a threshold as z-score=2.6 means that the value is above 99.5 percentile [4], effectively suggesting that the culture is distinguishable from other cultures with its association with the symbol.

[1] Generalization vs. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data

[2] Large Language Models Struggle to Learn Long-Tail Knowledge (ICML 2023)

[3] When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (ACL 2023)

[4] https://www.sjsu.edu/faculty/gerstman/EpiInfo/z-table.htm

Dear Reviewer PsWQ,

Thank you again for your review! We are happy to hear that you appreciated the importance of our research problem and the depth of our data.

Based on your thoughtful feedback, we wrote a detailed rebuttal covering the following points:

• Justification of Equation 1 and definition of memorization and generalization
• Clarification of the purpose of studying symbol and culture overmemorization
• Clarification of definitions of symbols and cultures

We would love to hear your thoughts about our rebuttal, including whether it sufficiently addresses your concerns and questions. If you believe that our rebuttal is satisfactory, it would be great if you could consider increasing your score. Any feedback is welcome and greatly appreciated!

Sincerely,

Paper13342 Authors