Dissecting Causal Biases

Overall, it is not clear to me what is learned from the paper. While the closed-form solutions are novel, they can be easily derived using existing causal inference tools. Further, there is little evidence supporting why these results are useful. Experiments could have strengthened this case, but the experimental section is toyish and it does not illustrate any real-world use-case. Lastly, the results in the paper crucially rely on the knoweldge of the causal structure, an assumption which is very unlikely to hold in practice, making the relevance of the paper for practicioners even more questionable. For these reasons, I'm not convinced that the paper makes an acceptance-worthy contribution.

There are a couple of minor typos. Figure 5 is not referenced in Theorem 3.1 so that readers may be confused about what are the variables of $\alpha$, $\beta$, $\gamma$. And please see [Q1] for detailed questions. Fonts in Figure 10-Figure 15 are too small and hard to recognize.

This paper is well written. It strikes me as more of a review piece summarizing existing literature or approaches than a stand-alone independent research contribution (e.g., the derivations under linearity are relatively standard; the class of biases identified are in general well-recognized within the causal inference community). In it's current framing, I think the paper has a place in the literature, but perhaps at a different venue (perhaps something like the Journal of Causal Inference, where review pieces are more typical, although I appreciate the motivation of the paper to "bring causal inference ideas to model training" [my summary]). The paper could also bolster its independent contribution by further focusing on the interaction between bias types (e.g., how selection and confounding bias interrelate under linearity).

I have a few more specific comments.

(1) The examples in Figure 1-4 are helpful, although the selected examples of the different bias types don't always feel the most natural (i.e., would have to be 'argued for' in the sense that different DAGs seem relevant (i.e., in certain situations it could be that labor union activism affects political beliefs through information channels, etc.).

(2) The paper is framed partly in terms of "let's bring causal inference ideas to model training" [my summary]. This is a reasonable framing, although the examples and the analyses sometimes don't focus as much on model training as much as generic causal inference questions. Therefore, I as a reader was left somewhat confused about the contribution (i.e., the abstract suggested more of an emphasis on the "causality for ML training" aspect, but the text seemed to focus on causal biases in a more generic (and therefore perhaps less novel) sense.

Overall, this paper is useful in its succinct and clear summarization of various causal biases, but has limitations in terms of broader contribution and framing.

• [W1] The transitions between the different sections of the paper are not smooth. There's no clear outline. First, the four types of biases are presented, then they are explained in more detail (close forms), and then they are approached by the modeling and experimental perspective. This sequence is only clear after the paper is read. I would suggest a more extensive outline of the structure of the paper somewhere in the introduction so the reader knows what to expect. I would also suggest adding some motivation to support the existence of the different "macro"-sections (those mentioned in the previous sentences).
• [W2] The presentation and difference of biases, discrimination and measurements is not clear, which makes difficult to understand what the purpose of the work is. In the introduction authors dedicated a full paragraph to make a distinction between discrimination and bias, but no clear connections are drawn between the two of them. This right after they motivate their work by saying "Addressing the problem of discrimination involves two main tasks. First, measuring discrimination as accurately and reliably as possible. Second, mitigating discrimination." But a few lines below, the authors state that "The main contribution of the paper is to use tools and existing results from the field of causality to generate closed-form expressions of four sources of bias." Given this, a clear connection between biases and the measurement of discrimination is needed.
• [W3] The results are presented in a non cohesive way inside the more theoretical sections. Some theorems are presented without a clear motivation. [W4] The paper is not very connected nor consequent: the work focuses in presenting closed form expressions of the different types of bias. They back up the importance of these formulae by saying that measuring this in the data is crucial to mitigate discrimination. Once they reach the experimental part, they show results on these measurements but do not reference the actual equations used for this. They also do not try to show that the assumptions of the theorems are valid.
• [W5] One of the most important points that should support this work (experimental validation) is shown from Figure 10-15 but these plots are not easy to read. The text of the legends, labels etc. are very small.
• [W6] Concepts such as confounding bias, sampling bias and intersectionality have been explored. studied and discussed by prior work. Little connection is drawn to this. For example, in the introduction paragraph on confounding bias (page 2), there is no citation to prior work. Similar holds for sampling bias

Further comments:

• Much of the theoretical sections is built from the linear regression coefficients. I would suggest adding a more detailed description of it and how its implications before presenting those sections.
• In section 5, it is written "Consistent with previous work," with no citation shown.
• Theorem 5.1 introduces is built around an Eq. (11) where no equal sign is written, which seems an odd way of defining a quantity (MeasBias).
• In section 7, the intro of that section is rather confusing: "The aim is to identify the cases where a given estimation of discrimination is biased and at which...". The aim of the authors' work? The aim of that section?

-Typos: -- Second paragraph of the introduction has an 'etc..' (missing a dot?). -- ConfBias is presented in different fonts. -- In addition, the considered causal structures mots? often show.