Data Checklist: On Unit-Testing Datasets with Usable Information

Thank you for your review and valuable feedback.

Baseline papers that look at the quality of data going into LMs

The discovery of annotation artifacts is typically evaluated qualitatively rather than quantitatively, even in well known papers like Dataset Cartography, since the existence of a particular attribute is a binary outcome that is hard to quantify in a granular way. We compare our findings with prior work on discovering data artifacts and show that our checklist can rediscover known artifacts.

Sec 5 has only two datasets and does not have enough analysis.

Data filtering is indeed a very exciting area, but since the focus of this paper is to formalize the data checklist taxonomy, data filtering is presented as a possible application—but not the main focus. However, we will consider experiments on other datasets, such as TinyStories for language modeling.

Thank you for pointing us to the NeurIPS paper on data selection! We will make sure to include it as a reference.

How is the $\epsilon$ chosen and more clarification on table 2.

As noted in Table 1, in each of the set of tests presented, we can choose either the $>\epsilon$ or $<\epsilon$ version as our assumption to test (chosen as viability, applicability, sufficiency, exclusivity, and necessity in Table 2). $\epsilon$ is the tolerance you have for test failure. Because we are ultimately taking an estimate of the V-info, even if the true V-info is 0, our estimate may be slightly above or below that. This is why $\epsilon$ is set to a small but non-zero value of 0.01. In practice, the user may also want to use values that are much larger than zero to satisfy some legal or product requirement (e.g. the gender attribute may be used to predict Y, but only to a certain extent, say 0.25 bits of information).

The implications of the table 2 scores are explained on page 6, but to go into more detail, take the sufficiency test as an example; it asserts that “X contains no more than 0.01 bits of usable information about Y that is not already contained in the length of X”. The test score is computed as $I(X -> Y | length(X)) = H(Y| length(X)) - H(Y| length(X) U X)$. Since this score is 0.036, which is greater than 0.01, the test fails, giving us the implication that the length alone is not sufficient to predict Y.

Nit: We will make sure to fix the references in the camera-ready.

We hope this response offers some clarification and, if it does, we hope you consider increasing your score!

Thank you for your review and valuable feedback!

The authors claim that they provide a library for easy creation of checklists for text generation as well as classification. However, they do not give the library link for the practical evaluation.

We did not include the link to avoid violating anonymity rules; we will release the code and include the link to the library in the paper after the anonymity period. Meanwhile, feel free to take a look at the supplementary material that we uploaded with the paper, which includes the code.

Q: The title does not reflect the content precisely.

Thank you for your suggestion! We’ll consider the title you suggested.

We hope this response offers some clarification and, if it does, we hope you consider increasing your score!

Thank you for your review and valuable feedback!

The paper presents too many different “contributions”.

We believe that the contributions we present in the paper are closely related. Building upon prior work in V-usable information, we propose a test taxonomy for checklisting datasets, and we then present experiments as examples of how people can utilize it, and show that this checklist is indeed useful. Finally, we extend this idea and take a step further to show how the checklist can be used for data filtering. If any one of these contributions were left out, it would become a purely theoretical or purely empirical paper, but it is a work that spans both. Given an additional page for the camera-ready, we will include more contextual and clarifying information to make the paper read more smoothly.

which range of values is expected for each test?

Great question! We will make sure to clarify this in the paper for the camera-ready.

Where |v| is the size of the vocabulary, the domain for all tests is [0, log |v|).

V-information is formulated as the difference between two entropy values (negative log probabilities), $H_V(Y)$ and $H_V(Y|X)$, where $H_V(Y) >= H_V(Y|X)$ since having access to X should never make Y less predictable (due to the optional ignorance property of the model family V). Thus we have a lower bound of 0. The upper bound of $H_V(Y)$ is the entropy of a perfectly uniform distribution over tokens, which is $E_{y} [ - log f$ O $(y) ] = log |v|$.

We hope this response offers some clarification and, if it does, we hope you consider increasing your score!

Thank you for your review and valuable feedback!

The paper assumes a lot of knowledge.

Due to the page limitation, we weren’t able to include detailed background information on all the datasets and preference alignment techniques mentioned in the paper, and instead, we provide brief context and citations for unfamiliar readers. However, we recognize that a smoother introduction and more explanations of datasets and techniques will be particularly helpful among a larger audience beyond the LLM space, and will facilitate the understanding and adaptation of our method. We will make sure to accommodate this suggestion in the camera-ready version.

The discussion of prior work talks a lot about model checklists instead of finding dataset artifacts, the focus of the paper.

The major part (first 2 of the 3 paragraphs) of Related Work (Section 6) is indeed focused on dataset analysis and annotation artifacts, where we cite many prior works on data attribution, data curation, spurious patterns, etc.

In addition, we are promoting the direct evaluation of datasets themselves instead of tracing problems back from model behavioral evaluation, which is why we start the intro by talking about model checklists and comparing them with our method.

While the results show that the approach is doing something, there is no comparison to any other approach.

The discovery of annotation artifacts is typically evaluated qualitatively rather than quantitatively, even in well known papers like Dataset Cartography. Although there is no agreed-upon list of annotation artifacts in the literature, we compare our findings with prior work and show that we can rediscover known artifacts with the new framework (section 4.1). The response length artifact in preference datasets has also been observed in prior work (Singhal et al., 2023), and we formalize the observation with our checklist. Our new taxonomy builds upon prior work, and we present contributions from related methods in the paper, even though a direct comparison is not easily available.

Q: You never define word complexity.

We measure word complexity as the frequency of occurrence in a large corpus (as mentioned in that section), where less frequent words are regarded as being more complex. For greater clarity, we will use the term rarity instead.

Thanks again for your questions and comments. We hope this response offers some clarification and, if it does, we hope you consider increasing your score!