Continuous Indeterminate Probability Neural Network

Dear Reviewer 7gbu,

We sincerely appreciate your very insightful and detailed feedbacks. Here are our detailed responses to your questions.

Q1: You write "pushed this classification capability to infinity" -> what does this mean?

A1: It means that the model is very flexible to classification tasks, we can use the same neural network without changing the number of output nodes to classify arbitrary categories. For example, in Fig. 9, we used two random variables (4 nodes in the last layer) to classify into 2 or 10 categories. And we can also use same model to classify a very large number of categories without limitation.

Q2: The motivation for this work in the introduction is based on the IPNN, which is however a model the reader knows nothing about at this point in the paper.

A2: We apologize for the additional reference to our other paper, which may have disrupted your reading.

Q3: You write "VAE uses neural network as the approximate solution of decoder" -> What does this mean?

A3: It means that the decoder of VAE is a model, but the decoder of our CIPAE is a probability equation, that means we do not need to design the decoder.

Q4: Overall Section 2.2 is hard to understand, and needs a better example/toy problem to help the reader.

A4: Actually, the toy example in Section 3 is a very good toy example to CIPNN.

Q5: What are Observers? Why do you need to introduce both Observer 2 and Observer 3? What is the difference?.

A5: In CIPNN, the label can be understood as Observer 1, the model output can be understood as Observer 3. (Observer 2 is the case for IPNN.)

In addition, we'd like to answer this question from a high-level.

The world is understood through observers. In the case of a coin toss, the outcome cannot be known unless observed. That is, the ground truth is not able to be known, we only know the observations.

Given that the use of observers is unavoidable, and the imperfect observations maybe more general in the real world? (Perfect observations are the special case.)

Additionally, we do not need to restrict the observers, as they may have different ways of understanding the world. For example, in our coin toss example in Section 3, Observer$_3$ understands the outcome as a Gaussian distribution.

The questions now is: what truly matters?

The Change! While different observers may have different understandings of the world, the Change still follows the Ground Truth (albeit with some errors). Our proposed equation is designed to evaluate this Change.

Q6: Why did you choose that specific distribution in the right-hand side of the KL divergence?

A6: As we written in the paper, KL divergence is for solving over-fitting problem, and our modification is according to the analysis results in Fig. 8.

Q7: "In this section, we will focus on the training strategy of Gaussian distribution" -> Can you clarify what this means?

A7: It means that using a distribution other than Gaussian, such as a uniform distribution or others, is also acceptable.

Q8: This section misses baselines for other classification models (even simple neural networks)?The classification performances of your model on MNIST look quite poor for example. In Table 3 you compare against "Simple-Softmax", which is not defined, and which performs significantly better than the proposed model

A8: Our proposed model is not for a performance improving, as you can see its performance is even poor than the most simple neural networks, so it doesn't need to compare it with other better models.

Q9: "As shown in Figure 5, the results of auto-encoder tasks between CIPAE and VAE are similar, this result further verifies that CIPAE is the analytical solution." -> What does this mean? Why can you make this statement from looking at a Figure?

A9: CIPAE is the analytical solution is a theoretical analysis results, and the results in Fig. 5 is an evidence.

Q10: "Although our proposed model is derived from indeterminate probability theory, we can see Determinate from the expectation form in Eq. (11). Finally, we’d like to finish our paper with one sentence: The world is determined with all Indeterminate!" -> not sure what this means.

A10: This is an abstract understanding of Eq. (11).

Best regards

Authors

Dear Reviewer Vwsb,

We sincerely appreciate your very insightful and detailed feedbacks, and we also thank you very much for your understanding of the unavoidable reference to our other paper. Here are our detailed responses to your questions.

Q1: Add some introduction to differences between the proposed model and VAE-like approaches

A1: OK. The framework of CIPAE and VAE are very similar, but the two models are derived from different theories.

1. The KL regularization term is an irremovable part in ELBO. Our KL regularization term is additional for over-fitting problem, and it is allowed to use other regularization method for CIPAE.
2. If we use a model as the decoder for CIPAE (not Eq. (1) in the paper), here is the detailed difference:

VAE:

$ELBO =-D_{KL}(q_{\phi}(\mathbf{z}\mid x^{(i)}) \parallel p_{\theta}(\mathbf{z})) + E_{q_{\phi}(\mathbf{z}\mid x^{(i)})} \left[\log p_{\theta}(x^{(i)}\mid \mathbf{z}) \right] \text{ (a)}$

Approximate CIPAE (Not the original CIPAE):

From (b) to (c): marginalization.
From (c) to (d): according to axiom 3 that given $\mathbf{z}$, $x$ and $y$ (image pixels) is mutual independent.
From (d) to (e): the decoder is approximated with a model $\theta$.

If we set the regularization term of CIPAE to be the same as that of VAE, our loss will be almost identical to that of VAE. The only difference is that the $\log$ term is outside of the expectation term $\mathbb{E}$. For the case where the Monte Carlo number C = 1, the losses of VAE and approximate CIPAE will be exactly the same.

This analysis results are quite interesting, that VAE and approximate CIPAE are derived from very different theory, however their final code implementations can be same in a special case (C=1).

Q2: Equation (21), being the basis for training, needs a deeper explanation. Why use the max functions both in numerator and denominator?

A2: Actually, the max functions are not important, because we usually set $\epsilon=10^{-20}$. Therefore, the max functions are not activated for the importantsample space.

Q3: ...the visualizations of the latent space become somehow different, with parts of the R^2 latent for VAE empty.

A3: You can get a sense of the difference by comparing Fig. 4a and Fig. 4b. In Fig. 4a, the effective points are confined to a small space, whereas the valid space of CIPNN is larger, as shown in Fig. 4b. Similarly, for CIPAE, although it is valid in a larger space, the values of Eq. (22) and Eq. (23) are very small, and the test or train points will not fall within that region.

Q4: From a practical point of view: what is the training and inference comparison between VAE and CIPAE?

A4: We are afraid that we cannot finished it during rebuttal. Maybe after rebuttal?

Dear Reviewer EMUS,

We sincerely appreciate your insightful feedbacks. Here are our detailed responses to your questions.

Q1: I find the motivation of this paper unclear. I had difficulty following the progression from Section 2.2 to Section 3 and then to the CIPNN model.

A1: We apologize for the additional reference to our other paper, which may have disrupted your reading.

Q2: The indeterminate probability theory is not surprising to me, and I believe it can be easily derived via the definition of conditional probability. Equations (1) and (2) hold for any , but it's not clear to me which specific type of z we are expecting in the learning process.

A2: At the very least, CIPAE has a very special property:: The famous VAE model requires a decoder model, whereas the decoder of CIPAE is a probability equation, and therefore does not require a model. Have you ever seen an image auto-encoder whose decoder does not require a model?

Q3: I find Proposition 1 to be weak from my perspective. Specifically, Proposition 1 states, ''If $P(y_l | z^1, ... , z^N) \to 1$, CIPNN converges to the global minimum.'' This is equivalent to saying that successful classification depends on our ability to learn a set of favorable variables, namely $z^1, ... , z^N$. However, the main challenge lies in determining the existence of these 'good' variables and how we can identify and obtain such $z^1, ... , z^N$ with theoretical guarantees.

A3: Your understanding is correct, the classification depends on our ability to learn a set of favorable variables, namely $z^1, ... , z^N$. Firstly, through experimentation, we can see that our models are able to learn these 'good' variables. Additionally, the main loss in Eq. (15) is an application of maximum likelihood which serves as a theoretical guarantee.

Q4: I did not find the comparison with existing approaches. Also, the numerical results did not indicates the improved performance is indeed from the introduction of $z^1, ... , z^N$.

A4: Our proposed model is not for a performance improving, as you can see its performance is even poor than the most simple neural networks, so it doesn't need to compare it with other better models.

Best regards

Authors