Sampling Process Brings Additional Bias for Debiased Recommendation

1. In theorem 3, the SuperDR approach’s generalization upper bound is kind of loose. What is the $K_{\phi}$? It's better to clarify the definition and interpretation of $K_{\phi}$.

2. What is the Rademacher complexity for this model. Provide a more detailed analysis of the Rademacher complexity for this specific model. What is the insight from this bound? Without a tight bound, practitioners may not be able to fully trust the model’s debiasing performance across various scenarios, limiting the method’s utility for diverse recommendation tasks where model generalization is critical. Discuss the practical implications of this bound for different recommendation scenarios. Compare this bound to bounds for other debiasing methods

3. Although large-scale datasets are used, the paper lacks a detailed discussion on the computational overhead and scalability of the proposed model in comparison to traditional DR models, which could impact its adoption. Authors can provide the runtime comparisons between SuperDR and traditional DR models on datasets of varying sizes or memory usage analysis or discuss of how the method scales with increasing numbers of users/items

1. The distribution problem of the sampling lacks practical significance. Because the debiasing methods also trained on the full dataset, $D$, rather than only training on observed data.
2. The improvement of some metrics is quite low and is not statistically significant.
3. Some definitions of symbols are missed, for example, $m$ used in line 210.

Some claims in the paper needs more clarifications: The paper claims " if the users and items in the training set are not exactly the same as those in the test set, all previous doubly robust based debiasing methods are biased". The paper’s statement oversimplifies the requirements for unbiasedness in DR estimation. In fact, DR methods do not require an exact match between the users and items in the training and test sets. It might ask for positivity and overlap of supports [1] although an exact match surely satisfy this requirement.

The paper claims the added correction has no harm, while it is theoretically correct in the case of the empirically covariance drops to zero. However, in practical scenarios, where exact zero empirical covariance is challenging to achieve due to model imperfections, 𝜖 may not fully converge to zero and could introduce slight adjustments, potentially affecting the "no harm" property.

The presentation of this paper can be improved: The lack of indexing in key formulas, especially when dealing with complex constructs like superpopulation, covariance terms, and probabilistic models, makes the paper harder to follow.

The paper has solid theoretical foundations, but it could benefit from more intuition and clearer explanations on why certain choices (like probabilistic models and covariance control) are necessary. The presentation would be stronger if it provided a more intuitive context for readers to understand how the superpopulation setting is motivated.

[1] Off-policy Bandits with Deficient Support, Noveen Sachdeva, Yi Su, Thorsten Joachims, KDD 20

W1: The writing of the paper seems somewhat hasty, with some key details missing and some descriptions challenging to understand. Here are some examples:

• In Section 1, critical undescribed content is why existing debiased recommendation methods must be unbiased in a train-test setting with the same set of users and items. Based on the current description, it is not intuitive.

• For Section 4.1, why, in a general scenario, does the learned imputed error or propensity estimate become the expectation? It is also not intuitive. In Section 2, the authors also mentioned that this work extends the previous methods for deterministic error imputation and propensity models to probabilistic ones. This is hard to understand.

• In Section 4.2, the imputation balance correction calculation process is unclear. For example, what does $m$ mean? Is $\epsilon$ a value? What are the meanings of the two terms of this imputation balance correction? Why does it achieve the desired control process?

• What is the meaning of the superscript Bal used in many losses?

• No description is provided for the division of the data set in the experiment.

• In Section 5.1, MF generates embeddings for each user and item, which are then used as their features in MLP. The rationality of this process is questioned.

W2: The motivation proposed may not be convincing enough. In real-world scenarios, it is rare to directly use a model trained on user (or item) set A to predict user (or item) set B. This is inherently risky because the embeddings of external users (or items) are completely random and unlearned. In fact, if SuperDR does not use a pre-trained recommendation model (such as MF) to obtain a set of well-trained user and item embeddings in the experiment, I think it will also encounter this problem.

W3: The core contribution of the proposed solution compared to existing work is introducing an optimization process that controls covariance. However, there is a lack of more in-depth analysis, and the overall contribution is not significant enough.

• In Section 4.1, the bias term consists of two parts. Although the first part (i.e., covariance) is repeatedly emphasized, it is unclear how the second part is optimized in SuperDR.

• From an intuitive point of view, why does controlling for covariance help correct estimates of external populations? What is the connection between them?

• Based on Algorithm 1, SuperDR does not make many major changes to the DR architecture. This makes people worry that it may not have much gain compared to DR in some cases. In fact, based on the results in Table 1, we can find that its gain is indeed marginal, especially on some larger datasets. In addition, the pre-optimization process for covariance seems difficult, which may cause unstable training in some scenarios.

W4: The current experiments are not sufficiently powerful and have many limitations.

• As described in the previous concern, the process of generating features for users and items is somewhat unusual.

• In real scenarios, there is much feature information besides user and item IDs. It is unknown how they can be applied to SuperDR.

• Considering that current recommendation systems contain many representative architectures, using only a simple MLP as a skeleton model is flawed and lacks applicability.

• Debiased recommendation is a hot research topic with many recent works. Some newer methods (e.g. published in 2024) should be included as baselines.

1. The most critical issue is that this paper assumes the users or items in the training set and test set are different. I don’t quite understand the rationale behind this assumption. Why does selection bias exist for newly added users/items that have not yet been interacted with?
2. The appearance of new users/items in the test set is essentially a cold-start problem. Could the authors compare their results with existing cold-start methods? Alternatively, could this approach also be plugged into cold-start methods?
3. Why is the dataset size so small? Why not use a larger dataset, such as the Amazon dataset or others?
4. In the experiment, when the sampling rate is not equal to 1, how are the embeddings for user/item that have not appeared in the training set handled? Are they initialized completely randomly?