No Free Delivery Service: Epistemic limits of passive data collection in complex social systems

Thank you for your thoughtful feedback. I agree that it is important to clarifying the prescriptive argument and the role of participatory methods. I hope that the response to all authors as well as the below additional clarifications can alleviate your concerns. Since everything is addressable via minor clarification updates, I hope this will also allow you to raise your score and confidence.

I’m ultimately not very sure what the prescriptive argument of this work is. I understand that there is value in work for pointing out a problem, but the issues with generalizing performance of ML systems into the real world is widely known and empirically shown (as demonstrated by the presence of the WILDS dataset [https://wilds.stanford.edu/datasets/]. This work provides a theoretical grounding for this, but from the abstract’s final sentence, I expected more implications rather than just gestures to participation and open science. I would have liked to see implications which demonstrate the strengths of approaching this problem from a theoretical perspective, as opposed to just the empirical evidence we have of it.

• Thank you for highlighting this issue, I agree that this is an important aspect. There are three aspects in your question that I would like to separate:
  • Prescriptive argument: The paper's theoretical results provide indeed actionable insights to improve data collection via the $k$-core condition for validity. In short, we would want to collect data such that either increases the $k$-connectivity or the size of the $\text{rank}(f)$-subgraph. Importantly, we can compute from a given sample graph where to collect data points. Please see the response to all authors for a detailed discussion of this.
  • Lack of justification versus impossibility results: If I understand your comment correctly, the key difference between the known results and issues that you are listing is that, to the best of my knowledge, these are based on a "lack of justification", i.e., that we know that our usual guarantees do not apply. In contrast, this paper provides much stronger insights in form of rigorous impossibility results. The insights that can be gained from such results are much more substantial and provide a path forward to improving the situation, as in the $k$-core condition above (while just noting a lack of justification can not). Please see the response to all authors for a detailed discussion of this (including also the different contributions of sufficient and necessary conditions).
  • Participatory: I agree that the current discussion of this is insufficient, thanks for pointing me to it. The argument for participatory methods stems from the sheer amount of data that would need be collected in a targeted way according to the $k$-core scheme above. While we are currently preparing a paper for submission that introduces a novel method for this purpose, I believe that it is beyond the scope of this paper to also cover this aspect in detail, (e.g., it would need to introduce additional concepts related to mechanism design, game design, economics, and the efficient computation of the $k$-core objectives). However, I will add a further discussion why participatory data collection is favorable for the setting of large-amount of data + targeted collection.

There is also increasingly a line of work on the limitations of using observational data (as opposed to data from, e.g., randomized controlled trials, as well as performative prediction, which I felt this work could connect to more, since that seems to be part of the premise of the issue with passive data collection.

• I agree that that this is an important aspect. However, I would point out that the paper actually discusses connections to work that goes beyond observational data, i.e., to counterfactual and causal estimators (e.g., see lines 261-265 in the submission). To the best of my knowledge this is also the first impossibility result for such estimators in settings as considered in this paper. As such, the theoretical insights provide, in my opinion, non-trivial insights into the limitations of these popular approaches. Thank you for pointing me to this. I believe it is an important contribution of the paper and will discuss it more clearly in the updated version.

A smaller point of confusion, I was unclear sometimes whether you were referring to “benchmark” only as the evaluation dataset or also the training dataset.

• I agree that this is not clear from the current presentation. Part of the issue comes from the fact that in (k-fold) cross-validation part of the training set act as validation set. I will clarify this (maybe in the appendix due to space constraints).

Thank you for your thoughtful feedback. I agree that W1 is important to clarify and hope that the response to all authors as well as the below additional clarifications can alleviate your concerns. Since everything is addressable via minor clarification updates, I hope this will also allow you to raise your score.

W1. While I enjoyed reading the formality and the definitions in section 2, I did not find the so-called no delivery service (NDS) of data surprising. As the author also stated, based on the theory of ML and the NFL theorem, when there is a sampling bias, and the source and target distributions are different, the expected performance guarantee of the model does not carry over. This has also been extensively discussed under "lack of generalizability" concept.

• Thank you for raising this important question. The key difference to the known results and issues that you are listing is that, to the best of my knowledge, these are based on a "lack of justification" argument, i.e., that we know that our usual guarantees do not apply. In contrast, this paper provides much stronger insights in form of rigorous impossibility results. The insights that can be gained from such results are much more substantial and provide a path forward to improving the situation (while a lack of justification can not). Please see the response to all authors for a detailed discussion of this (also in terms of sufficient versus necessary conditions and their different contributions).
• I also want to highlight that the results in this paper do point to an actionable path forward via targeted data collection using the $k$-core condition. This is also discussed in detail in the response to all authors. Again, this is possible because of the theoretical insights in this paper.
• For "lack of generalizability": How I am familiar with this concept in the context of validity theory is mostly as an empirical observation, e.g., that studies do not generalize beyond their very specific context. The results in this paper are again stronger, in the sense that the benchmark itself is invalid (not only that it doesn't generalize to new settings).
• The practical importance of the results is also highlighted by the wide-spread usage of benchmarks like MovieLens in settings where it cannot be valid, even in SOTA LLM benchmarks such as BigBench (https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/movie_recommendation/README.md)

W2. It is not particularly a weakness, but if I am not mistaken, the NDS observation for complex social systems naturally follows the previously known heavy-tailed distribution property of these systems.

• The NFDS results in this paper are two-fold:
  • They first establish necessary conditions that need to hold for model validation to be valid
  • and then show that these conditions are violated when sampling from complex social systems
• As such the necessary conditions hold independently of the concrete application in social systems.
• While the heavy-tailed distribution in complex systems have indeed been established prior to this work, one contribution of this paper is to combine this knowledge with the newly established necessary conditions to gain non-trivial insights into current practice of AI.

Thank you for your thoughtful feedback. I agree that your comments address important question and hope that the response to all authors as well as the below additional clarifications can alleviate your concerns. Since everything is addressable via minor clarification updates, I hope this will also allow you to raise your score.

The formal proofs and theoretical discussions might be too complex for practitioners without a strong background in the relevant mathematical and statistical concepts.

• Thank you for the suggestion. My aim is certainly to make this paper as accessible as possible, and I tried to do so via informal statements of the main results and providing extensive context. Unfortunately, the page constraints set limit on how much this is possible. However, I will follow your suggestion and add more intuitive examples where it is possible.
• While I acknowledge the issue, I'd also point out that the impression might not be fully uniform, e.g., feedback from other reviewers states that the paper is "easy to follow and fun to read", "very well-written".

Could you elaborate on alternative model validation paradigms that could be more suitable for complex social systems

• Thank you for raising this question. With my current knowledge, I would not so much point to alternative model validation paradigms but rather to improved data collection for model validation via the paper's $k$-core condition. For details on this, please see the response to all authors.

Could you provide more examples or case studies beyond the MOVIELENS benchmark to illustrate the validity issues in different types of AI systems

• Thank you for this question. Indeed, these results apply to a wide range of settings. In the PDF attached to the response to all authors, I illustrate this using widely used benchmarks for three different settings:
  • Reasoning - FB15K-237 (https://paperswithcode.com/dataset/fb15k-237): Here the goal is to infer the truth value of (subject, predicate, object) triples using logical reasoning. It is a widely used benchmark for reasoning and, as can be seen from the PDF, has the same structural properties as MovieLens. As such the results of this paper apply directly (also discussed in Section 3 and Appendix D.3). The social system the generates the biased observation is both the production of knowledge and its recording in FreeBase.
  • Link Prediction in Graphs - CORA (https://paperswithcode.com/sota/link-prediction-on-cora): Here, the goal is to predict links/edges in a citation graph based on observed edges. It is a widely used benchmak in graph learning and again shows the same structural properties at MovieLens. The social system that generates the biased observations is the citation practice in science (incl. popularity bias etc).
  • Extreme Classification - Wiki10-31k (http://manikvarma.org/downloads/XC/XMLRepository.html): Here, the goal is to predict the correct labels for Wikipedia entries from a large number of user provided labels (hence extreme classification). The social system that generates the biased observation are the authors that contribute Wikipedia entries and their labeling practice.
• In addition to these datasets, I also want to highlight that MovieLens is THE recommender systems benchmark (comparable to MNIST for vision) and is still widely used, even in SOTA LLM benchmarks such as BigBench (https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/movie_recommendation/README.md)

Could you discuss any potential limitations of your approach and how they might be addressed in future research?

• Please see the limitations section in appendix of the paper.

The results are heavily dependent on the specific characteristics of the social systems and data collection methods considered, which might limit the generalizability of the findings to other contexts or systems.

• The NFDS results in this paper actually are two-fold:
  • 1. It first establishes necessary conditions that need to hold for any model validation to be valid
  • 2. and then shows that these conditions are violated when sampling from complex social systems
• As such, the necessary conditions in 1) hold independently of the concrete application in social systems and provide general insights into the validity of model validation.
• That being said, I again want to highlight the urgent need to also understand the interaction with social systems since this is affecting much of our practice. Hence, even if the results would only be contained to this setting, I wouldn't consider it a limitation.