Multilayer Matrix Factorization via Dimension-Reducing Diffusion Variational Inference

We are much obliged to you for your careful review and constructive comments. We will carefully take into account your comments in our revision, and we would like to discuss various aspects as follows.

Regarding “Other Strengths and Weaknesses”

Point 1: Thank you for expressing your view in the beginning that “the use of diffusion VI is promising and interesting to me.” In Point 1, you mentioned that the improvement from the inference viewpoint may not be so significant.

Indeed, once you understand the principle of the equal-dimension diffusion models very well, it is perhaps not difficult to anticipate that dimensional-reducing diffusion models (more accurately, embedding dimensionality reduction directly into the diffusion model) could be constructed—at least intuitively. Still, the latter was not tried before. There are technical details to overcome, and some of them are not trivial. We would say that we start with a simple idea, but there are non-trivial details to work out, both in the development and in experiments.

Point 2: As with some fundamental research, at this stage we focus on developing a concept (specifically, a new inference technique for MMF) and providing proof of concepts. As future work, the application of the concept to more complex datasets could be considered. Also, the hyperspectral unmixing problem we demonstrated in this paper is an important representative application in the context of hyperspectral remote sensing.

Point 3: In addition to MSE, researchers in hyperspectral remote sensing also consider the abundance angle distance (AAD), which uses the angle of two vectors as the measure of similarity. We have provided AAD in the anonymous GitHub repository；please see Table 2 there.

Regarding “Other Comments Or Suggestions”

We agree and will try our best to better streamline the writing.

Regarding “Questions for Authors”

Point 1: Yes, optimization with semi-orthogonal matrix constraints is more difficult in principle. But that does not stop researchers from trying. In machine learning, for example,

[1] Moustapha Cisse et al. "Parseval Networks: Improving Robustness to Adversarial Examples", ICML 2017
[2] Nitin Bansal et al. "Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?", NeurIPS 2018,

researchers have found success with their numerical results. The regularization method we use is essentially the same as that in the above references. Our empirical experience with the MMF application is that the regularization method works reasonably.

Point 2: To answer your question, we provide some numerical results in Table 3 and 4. We observe that the $\boldsymbol{U}_t$'s are quite close to semi-orthogonality.

Concerning the question why semi-orthogonal $\boldsymbol{U}_t$'s are used, one important reason is to simplify the variational process. It can be shown that if $\boldsymbol{U}_t$'s are not semi-orthogonal, the variational operations would be more complicated. In fact, if we do not control the rank of $\boldsymbol{U}_t$ (which we do so via semi-orthogonality), there will be a lot of problems.

Point 3: In the context of MMF, our experience is that we do not need as many layer as in generative models, which can be thousands. We would say tens for MMF. The concept introduced in this work does not pose a constraint on the number of layers one can use, however, and as future work it would be interesting to see other applications that require more layers.

We appreciate your time and effort in reviewing our work, and we are grateful for your generally positive feedback. We will take your advice to improve the paper. We also want to discuss some of the main points you raised.

Regarding “Essential References not Cited”

We agree, and thank you for providing us with the references. We will add a paragraph in the Introduction to describe existing works related to dimension-varying diffusion.

We take this opportunity to give a reflection. The existing papers for dimension-varying diffusion are in the topic of generative models. They can be seen as concatenations of dimensionality-reduction processes and diffusion models. For the case of the references you provided, they concatenate multiple dimensionality-reduction processes and multiple diffusion models. On the other hand, we consider the topic of multilayer matrix factorization (MMF). We consider a single diffusion model and employ dimensionality reduction at each layer of the diffusion process, with the aim to develop a per-layer light-weight inference scheme. This makes the proposed method quite different from the dimension-varying diffusion methods we see in the prior generative modeling literature. However we agree that covering the prior literature would enhance the quality of this work, providing a broader coverage of related studies.

Regarding Comparison with Hierarchical VAEs

You mentioned that we didn’t give a lot of insights and justification on how diffusion-model based MMF differs from hierarchical VAEs (HVAEs). That is a good question, and we will try to improve the writing on this part in the revision. Initially, when we wrote this paper, we planned to cover the HVAEs in the main development (Section 3.3). However, we ended up abandoning it due to space limitation. It is in fact possible to apply HVAEs to the MMF model, associating each layer of the factorization model with one deep neural network for variational inference. We are not aware of any work on HVAEs for MMF, although it is possible. As described in [1], the stochastic approximation in HVAEs may have larger variance when the number of layers is larger. This, together with the need to use one deep neural network for each layer of the factorization model, may not fit the purpose of per-layer light-weight operations well. One advantage of the diffusion model is that it uses a simple variational (diffusion) model to greatly simplify the variational process and sidestep the aforementioned high-variance issue [1]—and this is our motivation for using diffusion models.

It is also worthwhile to note the following: In principle, HVAEs can be directly applied to MMF. But the previous diffusion model cannot, because it assumes equal dimension with the latent variables. Our endeavor, simply speaking, is to make the MMF application possible.

[1] Luo, Calvin. "Understanding diffusion models: A unified perspective." arXiv preprint arXiv:2208.11970 (2022).

Regarding “Other Comments or Suggestions”

We see your point and will try our best to streamline and make the writing more concise. The work consists of heavy details at some points, which can be better balanced, though we want to say that our writing style is to make the underlying assumptions and tricks clear to the reader; once again, we will try our best to balance. On “section 3.3.3 can be expanded,” you made a very good point and we agree. This, however, will be left as future work. We believe that from there we can develop more results.

Regarding “Questions to Authors”

Q1: We have run some simulations during this rebuttal period. Please see Table 1 in the anonymous GitHub repository. So far, our empirical finding is that, in general, the HVAE does not lead to performance improvement compared to the VAE. A possible reason is that the HVAE may require more careful training skills and more computations to achieve good performance.

Q2: One future direction is to perform further analysis to expand or better understand the diffusion-based MMF interpretation in Section 3.3.3. We think that this part is quite unique to MMF in terms of providing interpretable explanations.

We are very thankful for your thoughtful comments. We will do our best to revise and improve the paper. Here we would like to give a reflection on the main points you raised.

Regarding “Essential References not Cited”

Thank you for pointing out some references related to dimension-varying diffusion. We agree that we should cover that, and we plan to add a paragraph in the Introduction to describe the related works and how the current work differs.

Let us take this opportunity to give an explanation. The related papers consider the context of generative models and generative recommendation models. They, from a high-level perspective, can be seen as concatenations of dimensionality-reduction processes and diffusion models. Dimensionality reduction is done outside of the diffusion process. Our proposed method considers the context of multilayer matrix factorization (MMF). We embed dimensionality reduction inside the diffusion process. Each layer of the factorization model (or each layer of the neural network) is associated with one layer of the diffusion process. These two features are not seen in the prior literature in generative models, to the best of our best knowledge, and are the unique parts of our development. In the context of MMF, the two features give us the opportunity to develop a per-layer light-weight scheme for MMF inference.

While there are differences between the prior works and our current work, both fundamentally and in the topic of interest, we look back and agree that we should cover the prior papers in dimension-varying diffusion.

Regarding “Questions to Authors”

Q1: Absolutely, we can provide the codes. You can find the codes at the anonymous GitHub repository.

Q2: It could be difficult to give a reason when different methods use different models. MiSiCNet considers spatial correlations in its model, and this may give an edge to MiSiCNet in some instances.