Imputation for prediction: beware of diminishing returns.

We thank the reviewer for their time and feedback.

We appreciate the suggested clarifications for various concepts and have incorporated them into our revised manuscript, along with corrections to the identified typos.

We thank the reviewer for their time and feedback.

TL;DR: Our keys findings have not been described empirically before. The experiments in [Le Morvan 2021] are on synthetic data and can only serve as illustrations of their theory. They cannot support general practical conclusions, and they do not even reach the same conclusions as ours because of a narrow synthetic focus.

The authors conclude that both better imputations and missingness indicators improve prediction performance. Neither of these findings is particularly surprising [...]

Our results actually differ from previously published findings, including Le Morvan 2021. Our conclusion is that the impact of imputation on prediction is very small, rather than “better imputations improve prediction performances”. The heart of this work is in the estimation of the effect sizes, and observing that in some important cases (powerful models + mask), better imputations do not even improve predictions.

Le Morvan et al. (2021) in their Figure 4 already demonstrated that the performance of such an oracle > imputation via chained equations > mean impute.

As the reviewer points out, Le Morvan et al. (2021) highlight in their Figure 4 a rather strong positive effect of imputation on prediction, as the prediction performance satisfies: mean imputation < MICE imputation < oracle imputation. This actually contradicts our main message, according to which the effect of imputation accuracy on prediction accuracy is very small. This contradiction is because their setting is favourable for imputation to improve prediction:

• The different imputations are combined with a MLP only => this is the least powerful model in our study, for which we showed that the potential benefit of imputation is greatest.

• They work on synthetic data, where X is Gaussian, and $Y = f(w^\top X) + noise$, with f the square function for example. Although it is not a linear function, it has a specific structure as it is a simple transformation of a linear function, and it is probably closer to our linear setting than to our real data setting. => In our study, linear outcomes yield larger effects than real outcomes.

• Since their data is Gaussian, the imputation task is easier than with real data, leading to large gaps in imputation performance (mean vs oracle).

These points show that their experiments correspond to our most favourable setting for imputation to have an impact on prediction (and even more favourable since they use Gaussian data). We show that in more realistic settings (powerful models + real data), the effect of imputation is greatly minored.

Given that both key findings are consistent with established theory and have been described before empirically, what are the novel contributions that would justify acceptance of the paper?

Thanks for this important question. The important novel contributions are:

• Synthetic data illustration versus Extensive real-data benchmark The main contributions of [Le Morvan 2021] are theoretical results, their empirical results are very limited (only synthetic Gaussian data, with a MLP as prediction model, one missing rate). They cannot support empirical conclusions as our work does.

• Different empirical conclusions Because they focus on a synthetic and favourable setting, they report an effect of imputation on prediction that is very optimistic compared to what can be obtained in reality, as we show in our study.

• An asymptotic theoretical result does not necessarily hold in practice in finite samples [Le Morvan 2021] conclude “In further work, it would be useful to theoretically characterize the learning behaviors of Impute-then-Regress methods in finite sample regimes.”, as their main theorem (Theorem 3.1 - Bayes consistency of Impute-then-regress procedures) is asymptotic. This work answers this question with a rigorous empirical study. The fact that our key findings are consistent with established theory is a result, because we did not know what to expect in finite samples.

• Identification of the factors that modulate the importance of imputation for prediction Contrary to existing works, we show that more powerful models, appending the mask, and non-linearity of outcomes, all tend to decrease the importance of imputation for better prediction. This is important as focusing on too narrow experimental settings, or averaging over all settings, do not allow to draw actionable and consistent conclusions.

• Mask in MCAR Concerning the result that the mask improves prediction even under MCAR data, [Le Morvan 2021] did not report this as a result. The effect of the mask in MCAR with MICE is unclear in their Figure 4, and again the experiments are too limited to allow for empirical conclusions.

Please let us know whether our message and positioning relative to [Le Morvan 2021] makes more sense to you in light of these clarifications. Thanks again for your time.

We thank the reviewer for their time and feedback.

I suggest exploring the correlation between reconstruction error and gain from adding the indicator.

Figure 14 in the updated pdf shows the effect of imputation accuracy on the gain from adding the indicator. We see that most effects are negative, indicating that using the missingness indicator brings the largest boost in prediction performance when imputations have low accuracy. Moreover, effects are strongest for the MLP and smallest for XGBoost, meaning that with more powerful models, prediction boosts due to the missingness indicator are less pronounced. This is coherent with the prediction performances presented in Figure 1. In terms of correlation, the strongest association occurs for the MLP (20% missing rate), with a correlation of -0.5 on average across datasets, and a high dispersion (some datasets have a correlation close to -1, and 3 datasets have a positive correlation).

Additionally, an experiment with a random mask appended would demonstrate that this result is not just a product of a larger number of parameters in the model or some regularisation.

We conducted experiments with a shuffled missingness indicator appended (see Figure 15 in the updated pdf). The columns of the indicator were shuffled for each sample. This preserves the total number of missing values per sample but removes information about which specific features are missing. Fig. 15a demonstrates that using a shuffled missingness indicator harms prediction performance, except for XGBoost for which performances are unchanged. In contrast, the true missingness indicator improves performances (fig. 1). Furthermore, the shuffled indicator does not affect the relationship between imputation accuracy and prediction accuracy (fig. 15b), whereas the true indicator reduces the effect size (fig.4). These results confirm that the benefit of the missingness indicator is not due to a regularization or merely encoding the number of missing values. Prediction models effectively leverage information about which features are missing, even though under MCAR, this information is unrelated to the unobserved values.

Relation to Bertsimas (2024) Thank you for highlighting this work. On the theoretical side, Bertsimas (2024) shows that mean imputation is Bayes optimal while mode imputation is not, and clarifies the meaning of "almost all" in the theorem of Le Morvan (2021) regarding the Bayes consistency of Impute-then-Regress procedures. On the experimental side, they compare mean imputation to mode imputation (for categorical features) and MICE (for numerical features) in terms of downstream prediction accuracy. Their findings do not lead to a single conclusion, as the results vary depending on the experimental setting: real vs. (semi-)synthetic data, or linear vs. non-linear outcomes. In contrast, we employ multiple imputers to quantify the association between imputation and prediction accuracy, and identify how various factors modulate this relationship (e.g., downstream prediction model, use of the mask, linear vs. non-linear outcomes). Additionally, we explore the effect of using the mask, which was not considered in Bertsimas 2024.

We thank the reviewer for their time and feedback.

Line 140: what do you mean by a "universally consistent" algorithm?

It refers to a learning algorithm that achieves asymptotically optimal performance (i.e., minimal possible error) for all data distribution (X, Y) [Gyorfi 2002]. For example, partitioning estimates of the regression function, kernel regression or k-NN regression, are universally consistent.

Why is the 'semi-synthetic' data (top of page 5) 'semi'? Can you elaborate on that?

We call it semi-synthetic because the response $Y$ is synthetic, but not the design matrix $X$. Specifically, $X$ is derived from real datasets in our benchmarks, while the response variable $Y$ is simulated as a linear function of $X$.

Line 341 uses the word "nuanced" in a strange way - so strange I couldn't figure out what the sentence in which it appears actually means.

We changed the original sentence:

However, this observation should be nuanced by the size of the effects.

to this one:

However, this observation should be interpreted with nuance in light of the effect sizes.

We hope this clarifies the intended meaning: effect sizes should be considered to avoid overemphasizing the positive impact of imputation on prediction.

About imputer comparison

Perhaps these results should not be downplayed?

Thank you for this comment. We initially chose not to emphasize these results, as they are not central to our main message. It could also be argued that the mean imputers' poorer performance is expected. Based on the reviewer’s feedback, we removed one of the sentences that downplayed this result.

About the writing

We thank the reviewer for their advice. We fixed the acronym definitions, added the definition of the missingness indicator, removed some redundancies, removed some unnecessary words such as ‘really’ and ‘interestingly, and went through the manuscript to try to linearize sentences.

[Gyorfi 2002] A distribution-free theory of nonparametric regression.

We thank the reviewer for their time and feedback.

Regarding the weaknesses reported

The reviewer is concerned by the small effect sizes reported, and questions the statistical significance of some conclusions. We acknowledge that the reported effect sizes of imputation on prediction accuracy are small—this is indeed one of our key messages. The reviewer rightly highlights the need for statistical support for finer-grained claims (e.g., effects are small but non-zero). We confirm that our conclusions are supported by statistically significant differences (details of the tests provided in a separate comment):

• Small but non-zero effect sizes: Effects are significantly greater than 0 across datasets for less powerful models (e.g. MLP) but not for more powerful ones (e.g. XGBoost). Looking at individual datasets reveals heterogeneity: some datasets show statistically significant effects, while others do not. In an updated Figure 4 (see updated pdf), round shapes indicate coefficients significantly greater than 0 (one-sided T-tests, pval < 0.05, Bonferroni correction), while triangles denote non-significant coefficients. Overall, in the best case (MLP, no mask, 20% missing rate), the effect of imputation on prediction is significantly > 0 for 13/19 datasets. In the worst case (XGBoost, no mask, 50% missing rate), this holds for 6/19 datasets.

• Higher correlation with linear outcomes: Wilcoxon signed-rank test pval=1e-19. This translates into more coefficients significantly > 0 in the linear case than in the real outcome case.

• Larger effects without the mask: Wilcoxon signed-rank test pval=1e-10.

These findings are consistent across experiments and statistically significant, reinforcing our overall message about the diminishing returns of imputation.

Regarding the missing rates

In about a third of the benchmark datasets, the dimension is small such that 20% missing means just one missing feature.

Each entry has a 20% chance of being missing, rather than each sample has 20% of its entries missing, we will clarify this in the paper. Therefore the number of missing features in a sample varies according to a binomial distribution. This creates more diversity than forcing a 20% of missing values per sample.

Would larger missingness rates show more significant correlations?

We ran supplementary experiments at a 70% missing rate on all datasets (on top of the 20% and 50% missing rates), and added a new figure in appendix which plots the effects and correlations as a function of the missing rates (Figure 12 in the updated pdf) for the various models. We see that effects are larger for higher missing rates: this is particularly clear for linear outcomes, but less for real outcomes. This suggests that imputation matters more at higher missing rates, although for real outcomes and powerful models, these effects are still very small. By contrast, correlations decrease when the missing rate increases, i.e, the association is noisier (less likely to be significant).

Regarding feature importances

“use explainability techniques on these models trained with missingness indicators to see if the importance/weights of features drop when they are missing”

We computed feature importances using feature permutation (scikit-learn's permutation_importance). For each dataset, we identified the two most important features on the test set. For each feature, we then re-calculated its importance across the subset of test samples for which the feature was missing, and across the subset where it was observed. This allows comparing the importance of a feature when it is missing versus when it is observed. This experiment was conducted using XGBoost with condexp or missforest imputation, both with and without the mask.

Results are shown on a new Figure (Figure 13 in the updated pdf), and indicate that on average, a feature is half as important when imputed compared to when observed, with considerable variability (i.e., many features are 10 times less important when imputed, and some features remain as important when imputed). When a mask is used, importances drop significantly more with missforest imputation compared to when no mask is used (Wilcoxon signed-rank test p-value < 0.01). However, this effect is not observed with condexp imputation.

Does the correlation of prediction performance and imputation accuracy (e.g, Figures 3 and 4) also show any difference by imputation method?

A given imputation method gives very similar accuracies when repeated on the same dataset with a new sampling of missing values, therefore looking at the correlation between imputation and prediction per imputation methods is not informative: there is too little variability in the imputation accuracy.