A Pairwise Pseudo-likelihood Approach for Matrix Completion with Informative Missingness

Thank you for the detailed review of our paper.

Weakness

W1: Please see our global rebuttal.

W2: It is worth highlighting that their contribution is to remove ill-posedness through the appropriate assumption. However, it is necessary to define the term "ill-posed" properly and to prove why assumption 1 makes the missing value estimation a well-posed problem. Please let me know if I miss something at this point.

A: Here, ill-posedness refers to the non-identifability issue.The informative missing usually introduces an ill-posed problem not only in matrix completion but also in typical regression (see [14] and [20] cited in the paper). We would like to clarify that we didn’t claim the ill-posedness is completely removed. With assumption 2.1, the problem still possesses some identifiability issues as we mentioned in Section 4 (constant shift and scaling). We show that the proposed pseudolikelihood can recover the remaining identifiable equivalence class (at a near-optimal asymptotic convergence rate of the standard well-posed (non-informative missing) setting). The problem is still ill-posed, however with pseudolikelihood, we can recover some useful information from it. A contribution of our work is to provide a solid understanding of what is identifiable and can be recoverable reliably.

W3: Although the experimental results with real data are not promising, this should not be a reason for rejection, given the theoretical guarantees and other strong contributions.

A: Thanks for valuing our contributions. For the experiment results, we would like to add some additional insight. Our method targets the challenging MNAR setup where many existing methods fail. But, there is no free lunch. To avoid the bias caused by the MNAR, the proposed method in principle avoids using certain information in the data. Although it alleviates bias under complex missing mechanisms, one more crucial impact that we showed in our work is that the proposed method cannot identify shift-scale equivalence. In general, it is expected that the proposed estimator is less efficient compared to some existing methods under MAR or uniform missingness. This efficiency loss is a price that we pay for the robustness against the violation of restrictive missingness assumptions like MAR and uniform missingness. In other words, we expect our method to behave like the following. For uniform or MAR missingness, our method would be less efficient but remains competitive. For (strong/moderate degree of) MNAR under pairwise missingness, our method would perform better than many existing methods. More importantly, many existing methods would be inconsistent in general. In the experiments, our method performs robustly well for all three datasets, while a few alternatives perform very well in one, but very badly in another dataset. This aligns with the above explanation.

W4: Citations

A: Thanks for pointing it out. We will explicitly cite related works [23, 24] in the paper to support the claim “the nuclear norm regularization has been commonly used to promote low-rankness” and add a review paper [Hu] to it. General MNAR is ill-posed not only in matrix completion but also in regression [27, 30] and statistical inference [25] in general. A review paper [Tang] will be added too.

W5: Notation

A: We are sorry for the notation errors in the paper. Yes P, Pr, p and $\mathbb{P}$ are all for probability distribution. We will unify the notation to Pr in the paper. We will add the definition of the inner product between two matrices “$\langle A, B \rangle = \sum_{i,j} A_{i,j} B_{i,j}$ for any matrices $A,B \in \mathbb{R}^{n_1\times n_2}$” in the beginning of Page 5. We use N to denote the normal distribution. We will change it to $\mathcal{N}$ to make it more clear. Also, we will replace the notation $\phi$ with $p_{\mathcal{N}}$ for p.d.f. of standard normal distribution, to avoid notation abuse. We will add the definition of [n] as you mentioned in the beginning of Section 5.

W6: Minor comments

A: Thank you so much for the careful checking. We will make corresponding changes. We will add formal definition of nonnegative function: $s: \mathbb{R} \rightarrow \mathbb{R}^+$ in L106. We will add $Y_{ij} \sim f_{ij}(y\mid A,\phi)$ in L82. We will remind readers that $m_1$ and $m_2$ are the dimensions of matrix $A$ in the beginning of Section 5. Also, we will change the notation $T_{i,j}$ and $Y_{i,j}$ to $T_{ij}$ and $Y_{ij}$ respectively.

Questions

L88 and Appendix B

A: Yes. The independent means $(Y_{i,j}, T_{i,j})$ and $(Y_{k,l}, T_{k,l})$ are independent of each other for $i\neq k$ or $j\neq l$. We will modify $a(y)$ to $s(y)$ in Appendix B to unify the notation.

References

[Hu] Zhanxuan H., Feiping N., Rong W., and Xuelong L. (2021). Low rank regularization: A review. Neural Networks.

[Tang] Niansheng T., and Yuanyuan J. (2018). Statistical inference for nonignorable missing-data problems: a selective review. Statistical Theory and Related Fields.

Dear Reviewer CJYv,

Thanks for your appreciation! We will be sure to clarify ill-posedness and clean up all the notations in the revision.

Best,

Authors of Submission 14957

We would like to thank you for the constructive feedback and review.

Weakness: A bit more discussion on how the constants $\kappa$ and $\rho$ influence the right hand bound on equation (7) would be insightful.

A: Thanks for the suggestion. When $\kappa$ is small, it means there exists one pair of indices such that $E((Y_{ij}-Y_{i’j’})^2)$ (other parts of $E [Z^2_{i,j,i’,j’}]$ is positive) is close to 0, and it falls into the difficult case to identify either of them. For $\rho \in [1, \infty)$, when $\rho=1$, it refers to the uniform missing setting, which is the simplest setting to deal with. When $\rho$ gets larger, there is more variability among $\pi_{i,j}$ and so the bound in (7) is larger.

Q1: In assumption 2.1 is it required to have $s(Y_{ij})$ distinct at least for one pair $(i,j)$ and $(i’,j’)$? Otherwise we may end up with a situation where the assumption 2.1 is another variant of non-uniform missing structure scenario.

A: It’s correct that if $s(Y_{ij})$ are all the same, the missing mechanism is degenerated to non-uniform missing. But we don’t require that assumption that at least a pair of $s(Y_{ij})$ are distinct. Here the implication is that our method also applies to non-uniform missing scenarios. The fact that our method works for a wide range of scenarios is practically appealing.

Q2: Why do we need the constraint $|A|_\infty \leq a$ in the optimization described in (6)?

A: This constraint was used in our theoretical analysis, and is common in matrix completion literature such as [5] and [32] cited in the paper. Practically, the effect of this constraint is mild or non-existent, as it is common to pick a very large $a$. Indeed, some prior works might drop this constraint completely in the practical algorithm, which typically speeds up the optimization.

Q3: Please see our global rebuttal for question/weakness on numerical experiments.

Q4: One thing that comes out in Table 1 tobacco dataset is that the standard error of pairwise pseudo-likelihood is the second highest among all the methods. It may be interesting to check through the simulations if the standard error of pairwise pseudo-likelihood is in general remain high?

A: That is a good point. We actually included standard errors in simulations. There are error bars in Figure 2, but they are small and hard to recognize. We will make it more obvious in the revision. We also had a table in the Appendix showing the standard errors (Table 2 in Appendix C). The standard error of pairwise pseudolikelihood is small. The performance issue in Tobacco dataset might be because the assumption of the proposed method is not satisfied (if one entry is missed, the subsequent entries are also missed)

We greatly appreciate the effort you have spent reviewing our paper.

Q1: Could the author give a more detailed definition of informative missingness and a concrete example?

A: We used “informative missingness” to refer to what statisticians often called Missing Not At Random (MNAR) assumption [LR], in which the probability of missingness can depend on the value of the variable that is not observed. In ratings of movies, for example, people are more likely to watch movies that they may like due to unknown preferences and those preferences would also affect the rating, the missing probability is therefore likely dependent on ratings which may be unobserved.

Q2: Are the authors the first to consider this informative missingness setting in matrix completion? And are there any other methods to address informative missingness aimed at matrix completion? And How to evaluate the strengths and weaknesses of these methods?

A: Please see our global rebuttal about related works and theoretical evaluation. As for evaluating the numerical performance of these methods, we require independent sampling of validation data or known generating mechanisms. In the simulation study, we generate data under the condition satisfied for our setting, and we have the validation data to evaluate the completion errors. Our methods outperform other methods. In the Tabacco real data, we create the missing mechanism following the steps described in [2], which satisfies the conditions in [2] but violates the assumption in our setting, and our method still gives robust performance. Please see more details in the global rebuttal as well.

Q3: What is the difference between the pseudo-likelihood method and Bayes modeling?

A: The pseudo-likelihood is not a full likelihood that is used in conventional Bayesian modeling. Indeed, Bayesian modeling for the missing not at random settings will require parametric modeling of the full distribution of the outcomes and missingness mechanism, which is usually viewed as a daunting task. Because of identifiability issues, the result from Bayesian modeling will typically depend heavily on the prior used, which is another undesirable feature. The pseudo-likelihood used eliminates unknown $t_{ij}$ and function $s(\cdot)$, which allow us to focus on the estimation of the target matrix.

Q4: Please see our global rebuttal.

Q5: It might be challenging to effectively identify whether a scenario qualifies as "informative missingness" and apply this approach. For instance, on the Tabacco Dataset, the method does not outperform the SNN method. Is this result arise from this phenomenon/dataset that doesn’t belong to the “informative missingness” scenario? How to identify a phenomenon/dataset belonging to the “informative missingness” scenario?

A: Yes, you are right that it is challenging to effectively identify informative missingness / missing not at random (MNAR). Indeed, it is known in the missing data literature that available data is generally unable to distinguish between missing at random and among MNAR models [4, 5]. As in page 4 of [D], "It is impossible to test whether the MAR condition is satisfied, and the reason should be intuitively clear. Because we do not know the values of the missing data, we cannot compare the values of those with and without missing data to see if they differ systematically on that variable." Therefore, a fundamental limitation in any missing data analysis is that the missing mechanism has to be somewhat assumed. Given this fundamental challenge, our goal here is to adopt a flexible enough missing mechanism so that the methods still work for a wide range of scenarios to alleviate the concerns of mis-specifying missing mechanisms.

The separable assumption that we assumed indeed includes uniform missing, missing-at-random and the focused MNAR assumption [BH] as special cases, and has been argued as a general but workable assumption [34]. For matrix completion, the separable assumption that we use in this work is already fairly relaxed compared to those in the missing data literature, since each entry could have a complicated observation probability based on its location as well as a function on the potential observation (even though it could be missing). In terms of location dependence, it is allowed to be completely flexible. Among matrix completion methods (especially those with theoretical guarantees), our assumption covers the uniform missing and missing-at-random settings, which is the focus of most matrix completion literature. In other words, in terms of the applicability of the missingness assumption, our work is generally more applicable than many existing literature that assumes uniform missing and missing-at-random settings. When those settings fail, our assumption could still hold and our method will still have theoretical guarantee while others may fail.

As for the Tabacco dataset, we note that the way to generate missingness is adapted from the SNN paper. SNN is well-designed for the underlying missing mechanism. When one entry is missed in Tobacco dataset, the entries in the following period are also missed. This does not satisfy the assumption of our work. So it is not surprising to see our method perform sub-optimality. However, the performance of our method still remains strong. This example can be used to illustrate a situation where the assumption of our method fails.

References:

[LR] Little R J A, Rubin D B. Missing data in large data sets. Statistical Methods and the Improvement of Data Quality. Academic Press, 1983: 215-243.

[D] Allison, P.D. (2002) Missing Data, Quantitative Applications in the Social Sciences, Thousand Oaks, CA:SAGE.

[LR2] Little R. J. and Rubin D. B. (2019) Statistical analysis with missing data, Hoboken, NJ:Wiley.

[BH] Gomer B. and Yuan K.H. (2021) Subtypes of the missing not at random missing data mechanism. Psychological Methods, 26(5), 559-598.

Dear Yuanhong, Thanks for carefully checking our proof. Really appreciate! I will help to clarify the theoretical analysis. Yes the change of maximum and summation is a bit problematic. Indeed, instead of taking maximum in the definition of \xi_{n,g}, we can take the direct sum of these two components, which leads to bounding the summation of two summations in Lemma A.6. While the order of these two summations is the same, it will eventually lead to the same order bound.

Thanks Yuanhong. For Q1, we had a gradient zeroing out step "model.mu.weight.grad.data.zero_()" after each gradient descent step, it was deleted wrongly when we cleaned up the code for submission (while deleting the comments used for our own purposes). Please include it in your implementation. Hope that doesn't bother you too much. For Q2, we made it clear that our method suffers from constant shift and scaling when reporting error metrics like RMSE and MAE. Therefore, we did a simple linear regression on validation dataset to mitigate such bias. Other methods also rely on validation dataset for hyper-parameter tuning. You may view the constant shift and scale as hyperparameters of our method (in some sense), which are tuned on validation data. Thanks again for the careful check and valuable comments. Hope you enjoyed reading our work!