Neuron-Enhanced AutoEncoder Matrix Completion and Collaborative Filtering: Theory and Practice

Thanks for your comments. Our responses to Weaknesses and Questions are as follows.

Response to Weaknesses:

1. Yes, our method is slightly slower than AEMC. However, the accuracy of our method is higher than that of AEMC in all cases of the experiments. In many real scenarios, a slightly higher time cost is acceptable as we require high accuracy.
2. In fact, we have explained the motivation in Section 5.4. In Tables 2 and 3, the rating matrices are close to square matrices and hence cannot fully demonstrate the superiority of our method over classical matrix factorization methods. Therefore, in Table 4, we use a subset of MovieLens-1M, where the matrix size is $3706\times 500$. In this matrices, $n\gg m$. According to our theorems, when $n$ is larger (compared to fixed network architecture), the generalization bound is tighter. Indeed, as shown by Table 4, the improvement over matrix factorization becomes more significant.
3. This is due to the page length limitation. We have tried our best to include more important results in the main paper.

Response to Questions:

1. Yes, it is not a standard autoencoder. However, it is still a neural network to reconstruct the input (though partially). That's why we still call it autoencoder.
2. Thanks for raising the question. The reason we didn't discuss it is that when $m$ changes, the size of the neural network will change, which influences many terms in the error bound, e.g. the number of parameters, the missing rate, and $\Vert\tilde{\boldsymbol{X}}\Vert_F$. It is difficult to directly analyze the impact of $m$. If we fix the size of the hidden layers and the missing rate and assume that the latent dimension of the data does not change, then $m$ has little impact on the bound.
3. If $p_{ij}$ is close to zero, we may not observe the corresponding entry of the matrix. So in practice, $p_{ij}$ stays far from zero. Note that the bound is related to $\sqrt{\sum_{(i, j) \in S} p_{i j}^{-2}} /(m n)$, where there is a denominator $mn$. Thus, a few small $p_{ij}$ may not have a big impact on the bound.
4. No. The values in Table 1 are the relative recovery errors defined in the first paragraph of Section 5.1.
5. Thanks for the insightful comment. The possible reason is that when the latent dimension is too small, it will be difficult to train the two autoencoders of our method. Figure 3(c) shows that AEMC is not sensitive to the latent dimension if it is larger than or equal to 10. The optimization of AEMC is easier than that of our AEMC-NE.
6. Sure, we tested Sigmoid and Tanh but their performances are much worse than that of ReLU.
7. We tried both (7) and (9) and found that their performances are similar. This also verified that our method AEMC-NE is useful for both MCAR (missing completely at random) and MNAR (missing not at random).

Thank you very much for the positive assessment on our work. Our responses are as follows.

Response to Weaknesses: Thanks for pointing out the weaknesses. In the revised paper, we added more context about collaborative filtering to Introduction (e.g. paragraph 1). Actually, we have provided the time cost comparison in the original submission. It was shown by Table 6 of the appendix (Appendix E).

Response to Question 1: Yes, we actually tested other loss functions. To ensure consistence with the evaluation metric RMSE, we use the square loss. If the evaluation metric is MAE, we’d better use absolute loss. By the way, the square loss is the most widely-used loss in collaborative filtering and matrix completion. Our method can be extended to the scenarios of implicit feedback if we use negative sampling and other loss functions such as BPR loss (Bayesian Personalized Ranking, S Rendle 2012). We added this discussion to the conclusion section of our paper.

Response to Question 2: No. Our element-wise network can learn an activation function adaptively.

Response to Question 3: This is inevitable in the setting of transductive learning. When $|S|$ or $|S^c|$ is close to zero, the uncertainty for the training error or test error will increase significantly.

Response to Question 4: Thanks for pointing out it. This architecture was motivated by the fact that the dimension of the synthetic data is 300, the latent dimension is 10, and in practice we may never know the true latent dimension. Thus we use a larger latent dimension 30 in the autoencoder. Figure 2(c) also showed the effectiveness of the larger latent dimension.

Thank you so much for the very positive evaluation. Our responses to your comments and questions are as follows.

Response to Weaknesses:

1. Theoretically, according to our theorems, the size of the neural networks should not be too large when compared to the size of the training data. The number of the parameters of the two autoencoders should be smaller than the number of observed elements in the rating matrix. Empirically, we do not use deep autoencoders unless the number ($n$) of samples is much larger than the number ($m$) of variables (features). For instance, for the real datasets considered in our paper, the main autoencoder has only one hidden layer and the element-wise autoencoder also has only one hidden layer. For the synthetic data, because the number of samples (3000) is much larger than the number of features (300), as shown by the caption of Figure 2, we use three hidden layers in the main autoencoder and two hidden layers in the element-wise autoencoder. Figure 2(b) shows that our method is not sensitive to the width of the hidden layers of the element-wise autoencoder. In sum, when $n$ is much larger than $m$, we use more hidden layers in the two autoencoders.
2. The reason is that the average of $p_{ij}$ is about $\frac{|S|}{mn}$. We have improved the explanations in the revised paper. In Table 2, some values are missing because in the original papers of the corresponding methods, the datasets were not tested.

Response to Questions:

1. Yes. This has been explained by our response to the Weakness 1.
2. Thanks for pointing out this property. Indeed, the two neural networks are not necessarily autoencoders. Their architectures can be asymmetric. Our theoretical results apply to general MLPs.
3. It is possible but it depends on data. In the theorem, when using zero filling, $\Vert\tilde{\boldsymbol{X}}\Vert_F$ can be reduced significantly; when using mean filling, $\Vert\tilde{\boldsymbol{X}}\Vert_F$ is large. If mean filling cannot reduce the training error significantly, the corresponding overall generalization bound will be looser.
4. The reason we didn't discuss $m$ is that when $m$ changes, the size of the neural network will change, which influences many terms in the error bound, e.g. the number of parameters, the missing rate, and $\Vert\tilde{\boldsymbol{X}}\Vert_F$. If we fix the size of the hidden layers and the missing rate and assume that the latent dimension of the data does not change, then $m$ has little impact on the bound.
5. There are two reasons for this setting. First, in real datasets, the true latent dimension of data is unknown. Thus we do not expect to use the true dimension in the experiment. Second, we find that, as shown by Figure 2(c), a larger $\hat{d}$ often leads to a better performance.
6. For implicit feedback, we need to use negative sampling and ranking loss functions such as BPR loss (Bayesian Personalized Ranking, S Rendle 2012). Our theoretical results apply to any loss functions. We added this discussion to the conclusion section of our paper.

Thanks a lot for recognizing our work. Our responses to your questions are as follows.

Response to Question 1: Fig. 1 just shows an example. We can exchange the roles of items and users. As mentioned in Analysis C of Section 3.1, increasing $n$ (the number of samples) will reduce the generalization error bound and we hence propose to construct an autoencoder for the users when there are more users than items and vice versa. In Table 4, we reported the experimental results of a few baselines on a subset of the MovieLens-1M dataset. The matrix size of the subset is 3706x500 (3706 items and 500 users), where the autoencoder of our method is for the items. We see the improvement over the matrix factorization method is more significant that that in Table 2 (entire MovieLens-1M). Actually, if the autoencoder is for users, the performance will degrade. This verifies the impact of the roles of items and users.

Response to Question 2: As stated in Section 3.2, $Q$ can be obtained by a few approaches (Wang et al., 2019; Ma & Chen, 2019) such as Naive Bayes or Logistic Regression (Schnabel et al., 2016).

Response to Question 3: In $\sqrt{\frac{L_W^3 \xi_{\bar{d}}}{n}}$, $L_W$ is the number of layers of the network, $\bar{d}$ is the maximum width, and $\zeta$ (sampling rate, usually very small) is in the range $(0,1)$. Usually, we do not use a large $L_W$. Suppose $n=5000$, $L_W=4$, $\bar{d}=200$, and $\zeta=0.1$ (quite large), then $\frac{L_W^3 \xi_{\bar{d}}}{n}=0.256$, which is much less than 1. Indeed, it is possible that $\frac{L_W^3 \xi_{\bar{d}}}{n}$ is larger than 1 when $n$ is too small compared to the size of the neural network. But in practice, $n$ is usually very large.