Indeterminate Probability Theory

Dear Reviewer S3ce,

Thank you very much for your detailed feedbacks.

Q1: What do you want to say in Section 2? It seems that the example does not go beyond classical probability theory, i.e., all definitions, quantities and calculations are consistent with definitions and axioms in classical probability theory.

A1: The example in Section 2 employs our proposed candidate axiom 3, which serves as a helpful introduction to our theory. Our intention is to provide readers with a seamless transition into our theory by using this example.

Q2: What is new in your indeterminate probability theory?

A2: We suggest that you take a look at the simple coin toss example in Appendix B. If you can solve it using other probability theories, then our theory would not offer anything new.

Q3: Specifically, I am confused why eq. (4) must be 0 or 1 in classical probability theory. Could the authors give some references?

A3: The point here is that Eq. (1) in the paper is only applicable when Eq. (4) is 0 or 1.

Q4: The authors should give references, clear derivations or rigorous counter-examples when refuting something in classical probability theory, as it is based on rigorous mathematics.

A4: Our proposed equation does not refute anything in classical probability theory. The classical probability Eq. (1) is a special case to our equation.

Q5: How general your proposed theory is? For example, does your theory enable A and Y to be any kind of random variables, and can the two-phase protocol in your theory model any data-generation process?

A5: The generalization of our theory is a conclusion from our proposed new candidate axioms. Therefore, if even a single exception is found, it would not be appropriate to assert that the theory is general. Moreover, we have applied our theory to thousand dimensional continuous latent spaces without any special requirements for the variables or datasets. That is why we say that the proposed equation is general.

Q6: In page 5 the authors say "...Otherwise, Candidate Axiom 2 and Candidate Axiom 3 cannot both be true". In my opinion, it is strange to discuss the soundness of an axiom once it is proposed.

A6: We have a differing opinion on this point. Since the axioms are newly proposed, readers should initially approach them with skepticism. The soundness of new axioms should not only be discussed, but also given the highest priority for verification, as they serve as the foundation for a new theory. (For example, it is wrong to claim naïve Bayes independent assumption as an axiom.)

Best regards

Authors

Dear authors,

Thanks for your various replies to my questions. Before examining them and the paper again (two things I will only be able to do with the limited time I can allocate to discuss/review a specific ICLR paper), I would like to point out that my initial comments aimed at pointing out general problems of the paper, and what authors have provided are rather specific answers compared to this generality. My current opinion is that before being accepted, the paper would need to clarify many things:

• First, being crystal clear about the mathematical statements made in the paper. This is especially important for axiomatic and theoretic studies. Such a thing as "Not applicable if $A_j$ is continuous" will not really make it, especially as frequentist interpretation of probabilities (one possible interpretation of Equation (1)) can perfectly deal with continuous spaces.

• Second, and as the work intends to be axiomatic, be very clear about the terminology. As said in the review, the word "indeterminate" usually refers to coarse/missing/imprecise aspects, something that is not really discussed here. So there is a need at least to explain how the notion of indeterminacy here differs from the classical notions of indeterminacy.

• Third positioning itself from other works that are very close to the presented idea, such as noisy observations (note that noise can be defined instance-wise as well).

• Fourth, all of this (clear positioning, unanmbiguous mathematics for all critics/statements) are necessary elements of a paper advancing a new theory/framework, that should not be relegated to the appendices for the sake of space limitations. As authors indicate by referring very often to the appendix, this may be something that is not doable in a conference format, and rather than potentially making a fourth submission to any A/A* conference of their preference, maybe a full journal paper (or a book, if they really want to expose a new, full-fledged theory) would be more suitable. On one side it would at least give more time to reviewers to fully read the paper and enter into the examples, and on the other side it would allow the authors to make a full, complete exposure of their work.

• Fifth and not last, I definitely remain wary of an uncertainty theory where "Axioms 1, 2, and 3 cannot be falsified is that they have no exception", at least for two reasons: this would suggest a "perfect theory", which arguably cannot exists, and also as I've said, I think a theory about uncertainty should be falsifiable to some extent, so as to know its limits and be open to criticism.

To be clear, I am not advancing that there is no value in the current paper (there may well be), but if there is, the current presentation does not allow me to assess them clearly (in particular in comparison to the very, very vast literature on probability foundations).

Best

Dear Reviewer Xi4x,

Thank you very much for your detailed feedbacks.

Q1: For instance, P2 top, it is not true that one cannot apply Bayes rule in continuous setting, and it has been numerous, numerous times.

A1: We respectfully wish to remind you that you have misunderstood our point in P2 top. We merely stated that continuous variable is not applicable to Eq. (1), not Bayes rule.

Q2: it is not clear for the naive Bayes what exactly means $P(A^j=a^j_{i_j}|Y=y_l)$ being not always solvable?

A2: The coin toss example in Appendix B is an evidence to this point.

Q3: is it that the observational process is noisy, or that the obtained probabilities are ill-known and hence that one should consider sets of possible probabilities? ... if the main idea of the paper is to have $P(y_{obs}=y|y_{true}=y)<1$ ($y$ here can be either the output value or a feature value)?

A3: Your formulation does not align with our theory. Hence, we will adopt your idea to frame our case as follows: $P(y_{obs}=y|x_{k})<1$ and $P(y_{true}=y|x_{k})=1$. Our focus is solely on the indeterminate observation for the $k^{th}$ random experiment. And our objective is to utilize $A_{obs}$ ($P(A_{obs}=a|x_{k})<1$) to infer $y_{obs}$. More details you can find in Q5 of Reviewer MWjN.

Q4: Similarly, if indeterminate means ill-defined, then there is a whole literature about that (see, e.g., work following the book on Peter Walley on imprecise probabilities and similar).

A4: Thank you for sharing Peter Walley's imprecise probabilities theory. From our limited understanding, this theory differs from ours in that it represents probabilities using sets or intervals of values, rather than a single point.

Q5: I would equally question a statistical learning theory or more generally an uncertainty theory whose axioms cannot be the subject of tests and falsification?

A5: To deny a new axiom, all you need is to find a single exception, even a simple toy example would suffice.

Q6: All theories of uncertainty I know of that are a bit serious in terms of operationally are subject to falsifiability, and this especially true for probabilistic theories (see the Ellsberg paradox for a good example of attempted falsification).

A6: Thank you for sharing the Ellsberg paradox with us. However, this example differs from our case. The reason why Axioms 1, 2, and 3 cannot be falsified is that they have no exceptions.

Q7: since Softmax does not enjoy peculiarly good properties from a theoretical perspective,

A7: We agree that softmax properties are not good enough. In our applications, softmax is only used for IPNN, and we have encountered local minimum problems with this function, as outlined in Appendix E.7. However, our CIPNN and MTS forecasting method rely on Gaussian distribution, not softmax function.

Best regards

Authors

Dear Reviewer MWjN,

We sincerely appreciate your insightful comments, and we are excited to receive such in-depth feedbacks. We appreciate your high-level discussion on this topic, as well as the new perspective in which you summarized the theory in the 'Questions' part. Here are our detailed responses to your questions.

Q1: ... if it can be expressed as abstract probability space.

A1: Yes, we will provide a detailed response starting from Q3.

Q2: your own understanding ... the proposed probability theory simply ignores the auxiliary random variable while simultaneously assuming Axiom 2.

A2: We are surprised by your new understanding of this theory, and we agree with you. We will provide further details to support this understanding.

$$\underset{\text{(a)}}{\underbrace{{\color{Red}P^{\mathbb{A}}\left ( y_{l}\mid x_{n+1} \right ) }}} = \underset{\text{(b)}}{\underbrace{\sum_{\mathbb{A}}P\left ( y_{l},\mathbb{A}\mid x_{n+1} \right )}} =\underset{\text{(c)}}{\underbrace{ \sum_{\mathbb{A}}\left (P\left ( y_{l}\mid \mathbb{A} \right ) \cdot P(\mathbb{A}\mid x_{n+1}) \right )}} =\underset{\text{(d)}}{\underbrace{ \sum_{\mathbb{A}}\left (\frac{ {\textstyle \sum_{k=1}^{n}\left ( {\color{Red} P(y_{l}\mid x_{k})}\cdot P(\mathbb{A}\mid x_{k}) \right ) } } {{\textstyle \sum_{k=1}^{n}P(\mathbb{A}\mid x_{k}) } } \cdot P(\mathbb{A}\mid x_{n+1}) \right )}} = \underset{\text{(e)}}{\underbrace{\text{Eq. (15)}}}$$

From (a) to (b): marginalization.
From (b) to (c): candidate axiom 3.
From (c) to (d): candidate axiom 2.
From (d) to (e): candidate axiom 1.

Where superscript $\mathbb{A}$ serves only as an indicator to distinguish the two red terms. For continuous random variables, we change $\sum_{\mathbb{A}}$ to $\int_{\mathbb{z}}$.

As you have mentioned, if the auxiliary (or latent) random variables $\mathbb{A}$ is eliminated through marginalization, the system loses observer independence. On the other hand, if we do not eliminate it, the current probability theory stops at step (b). (Of course, approximation methods or analytical solutions for special datasets can handle it further.)

Q3: a situation where all randomness in the world is governed by an abstract space $\Theta$, where all randomness is lost if the abstract space is determined at a point $\theta \in \Theta$ , and where all variables can be described deterministically.

A3: We like your point and also such kind of high-level discussion. However, we are concerned that the point may have problem at the microscopic level according to 'quantum uncertainty'.

Our further opinion is that the world can be very good predicted if we have sufficient multivariate latent variables $\mathbb{A}$ (or $\mathbb{z}$). This can be expressed as: $P^{\mathbb{A}}\left ( Y\mid x_{n+1} \right )$

Where $\mathbb{A}$ is latent variables and $Y$ is our interested variables. By utilizing a sufficient number of historical observations of $\mathbb{A}$ adn $Y$, we can use this equation to make a prediction. Although this calculation is very complex, it is of polynomial complexity rather than exponential. Additionally, this theory can be applied to time series forecasting.

Q4: In the abstract space, random variables are represented as a projection of the world as a map to an object, e.g. $Y(\theta),X(\theta),A(\theta)$, can be uniquely determined for a given source $\theta$.

A4: Firstly, $X$ is a very special random variable, conditioning on $X$ is only used to indicate which random experiment is being referred to. We'd like to express the probability space as abstract variables $\mathbb{A}$ (or $\mathbb{z}$) and $Y$, and $P_{\theta}(\mathbb{A}|x_{k})$ and $P_{\theta}(Y|x_{k})$ can be uniquely determined for a given source $\theta$. Our applications of IPNN, CIPNN and CIPAE serve as good illustrations of this point.

Q5: Not your question but you may have interest. Why the auxillary observers are general?

A5: The world is understood through observers. In the case of a coin toss, the outcome cannot be known unless observed. That is, the ground truth is not able to be known, we only know the observations.

Given that the use of observers is unavoidable, and the imperfect observations maybe more general in the real world? (Perfect observations are the special case.)

Additionally, we do not need to restrict the observers, as they may have different ways of understanding the world. For example, in our coin toss example in Appendix B, Observer$_3$ understands the outcome as a Gaussian distribution.

The questions now is: what truly matters?

The Change! While different observers may have different understandings of the world, the Change still follows the Ground Truth (albeit with some errors). Our proposed equation is designed to evaluate this Change.

Finally, we are willing to have more discussion with you and look forward to your feedback.

Best regards

Authors