Quantum-Inspired Reinforcement Learning in the Presence of Epistemic Ambivalence

We would like to thank you for the precious feedback. Below, we respond to your questions in the same order. We sincerely hope that you will find the responses and revisions satisfactory.

Response to Q1: Our proposed EA-MDP framework can be extended in several ways. For instance, as mentioned in the paper, one could consider using a time-dependent quantum state (revoking Assumption 1), or a time-dependent outcome set. Another possible direction for future research could involve combining EA-MDP with PO-MDP or using non-stationary rewards. Based on your comment, we will add such possible extensions as future research directions to the paper.

Response to Q2: The main challenge in this context is to convert classical data into quantum data, particularly in the form of superposition. For example, consider multiple pieces of evidence that need to be encoded together with the corresponding probability amplitudes. In recent years, the conversion of classical data into quantum data (quantum states) has gained significant attention (Biamonte, et al. -2017). For example, mapping classical pixel values to quantum states has been explored in various applications (Yan,Venegas-Andraca-2020). In EA-MDP, there are more challenges than typical quantum machine learning. In addition to encoding classical data to quantum data with corresponding probability amplitudes, a set of outcome representations is needed. This set of outcomes is essential for calculating the rewards, as it is used to determine the measurement and the reward. Selecting such a suitable outcome set is crucial to ensure accurate reward calculations. In addition, in real-world experiments with non-separable outcome set, the number and dimension of outcomes may increase significantly. If $n$ is the number of the underlying states and $m$ is the number of EA bases, the memory needed to store the outcomes scales by a factor of $O(( m n)^2)$. Similarly, the memory needed to store the separated quantum states like Eq. (3) scales by a factor of $O( m n)$. Furthermore, the computational time required to calculate the reward also scales as $O((m n)^2)$. Based on your comment, we will add these challenges to the paper.

Response to Q3: Traditional RL methods are designed to perform efficiently in simpler environments under MDPs with less complex dynamics than EA-MDP. Therefore, RL agents trained with traditional RL algorithms struggle in EA-MDP-based environments when dealing with contradictory evidences. Compared to EA-MDP, traditional RL frameworks do not consider epistemic ambivalence into their model and solution. As a result, agents trained with traditional RL algorithms perform poorly in our setting and fail to converge. In contrast, EA-MDP framework deals with EA effectively by using quantum superposition and probability amplitudes. In turn, our proposed algorithm, which is designed specifically for EA-MDP-based environments, helps agents make informed decisions in the presence of epistemic ambivalence.

Dear reviewer eEvs.

We appreciate your time and work when reviewing our paper.

We hope that our replies appropriately addressed your concerns and explained the issues you mentioned.

If there are any remaining questions or require clarification, please let us know. We will be happy to give extra information and discuss them further.

Thank you for your time and for contributing to the review process.

Best Regards,

Authors

We would like to thank you for the precious feedback. Below, we respond to your questions in the same order. We sincerely hope that you will find the responses and revisions satisfactory.

• Response to weaknesses:

  • We are currently running additional experiments based on your comment and add them to the paper as we obtain results.

  • We are uncertain regarding the definition of entropy that the reviewer has in mind, as the entropy can be used to measure uncertainty in various components of our problem formulation. For example, in quantum mechanics, the Von Neumann entropy is defined as $S(\rho) = -Tr(\rho \log \rho)$ (Nielsen and Chuang, 2010). Since in our proposed EA-MDP formulation we used a pure quantum state, the density matrix is defined as $\rho = |\tilde{\boldsymbol{s}}> <\tilde{\boldsymbol{s}}|$. As a result, the Von Neumann entropy is always zero. This is due to the fact that pure states do not have mixedness. As another example, in policy optimization, state entropy can be calculated which is defined based on the current policy, i.e., the probability distribution over actions, to measure the uncertainty of decision-making for certain states. We hope that the reviewer can provide us with some references to make the discussion on entropy more explicit. Regardless of the specific entropy definition, we will calculate the policy entropy in our experiments and add the results to the paper based on your comment.

  • Based on your comment, we will move most of the equations in Section 6 to the Appendix to have more space to explain the experiments and the intuition behind them further.

• Response to Q1: By comparing (8) and (11), we conjecture that (left side) = (right side) in (12). To make it more clear, we will replace $\cong$ with equality sign (=).

• Response to Q2: In the first experiment, we used a single action to make the state transition probability and the policy deterministic. This approach allowed us to focus on analytically calculating the value function based on the computed rewards, which in turn helps to clarify the EA formulation and validate the theory. However, as an alternative, we could consider a more general setting with two actions in each site, one that moves to the other site and another one that remains in the same site. In this case, we could assume that the corresponding state-transition probability of remaining at the same site is zero. This alternative would be equivalent to our current setting. In our experiment, we defined a single action to avoid confusion and simplify the setting. In any case, we assumed so in order to make the state transition probability and policy deterministic.

I thank the authors for the updated draft of their paper. The new experiment that compares the EA-MDP to max entropy RL has increased my confidence that the proposed method has merit, and has significantly strengthened the paper's claims. However, I have chosen to keep my original score for the following reasons:

• Above all else, I still feel that this paper can significantly benefit from another round of reviews. In particular, it is still not clear to me how I can take a given decision-making problem and convert it to an EA-MDP. For example, how would I select my bases and their amplitudes? Is it an arbitrary choice, or is there logic involved? Can any decision-making problem be converted to an EA-MDP? These are the kinds of questions that the current draft of the paper does not answer (note that this concern was mentioned in my original review). It appears that the authors have all the pieces in place for a compelling paper, but they need to further refine the paper's narrative to communicate the results in an adequate manner.
• Similarly, while the comparison to entropy is encouraging, the authors do not provide any commentary on whether it makes sense to compare the EA-MDP to entropy to begin with, or how the comparison highlights the strengths of their approach. Ideally, the comparison should highlight a weakness in max entropy methods (or other baselines) and show (with an accompanying discussion) how and why the EA-MDP overcomes such weaknesses. To tie this back to my first point, one potential narrative device that the authors could explore is to first present an experiment that highlights a weakness of max entropy (or other baseline), then use it to motivate the EA-MDP, show how one can convert the max entropy MDP to an EA-MDP, then show the comparison. All in all, the authors need to better communicate how and where their methods fit in relative to similar RL methods.

I thank the authors for their commitment and effort, and I hope that these suggestions will help improve the paper moving forward.

We thank the reviewer for the recent comments. In the following, we will clarify the intuition behind our proposed method and its general workflow based on the reviewer's concerns.

Comment: The narrative of the paper suggests that one should be able to take a given MDP-based system that has epistemic ambivalence, convert it to an EA-MDP, and then use the proposed quantum-inspired methods to find an optimal policy.

Response: We would like to clarify that, such conversion of MDP-based system that has epistemic ambivalence to EA-MDP is not the intent of our proposed method. As correctly pointed out by the reviewer such misunderstanding could potentially arise due to the fact that one would possibly distinguish between an MDP-based system that has epistemic ambivalence and an EA-MDP. However, these two concepts are the same, and we do not distinguish the two in our paper. In fact, to motivate the idea of EA-MDP and its connection to MDP, our paper starts by defining MDP, then combines it with EA and calls it EA-MDP. Hence, to emphasize, an MDP with EA is the same as EA-MDP. Nevertheless, based on your comment, we will make these points more clear in the final version, provided that we can obtain the editor's agreement.

Comment: As per the authors' most recent comments, there are several conditions that need to be met in order to use the EA-MDP, such as having to convert the data to quantum data. This is something that is not clearly communicated in the most recent draft.

Response: We would like to bring to your attention that we already included these points based on reviewers' comments in the revised paper and highlighted them as limitations and challenges of implementing EA-MDP in practice, including the necessity of converting the classical data to quantum data (please see lines 1117-1129).

If you have any remaining questions or require clarification, please let us know. We will be pleased to provide more information and discuss it more.

We would like to thank you for the precious feedback and positive evaluation of our work. Below, we respond to your questions in the same order. We sincerely hope that you will find the responses and revisions satisfactory.

Response to the first part of Q1: In our formulation, to preserve generality, we propose a general formulation for EA-MDP and consider the general form of an outcome. However, in Section D of the supplementary material, we propose a simplified version of our formulation which is based on separated outcomes and define the reward based on the EA part of the outcome. Consequently, as correctly pointed out by the reviewer, in our experiments, we work with this simplified version and consider only the EA part of outcome. This simplified version is sufficient for defining the outcomes based on the quantum states introduced in Eq. (3). Hence, we worked with the separated outcomes in our experiments in order to execute the proposed EA-MDP in practice while keeping the setting simple for the non-technical readers. However, in a more complex experiment, one may consider non-separable outcome sets.

Response to the second part of Q1: In Eq. (3), we consider two subsystems for a given quantum system. One for the underlying states and another one for the EA quantum state. Each $| \psi \rangle$ is the mixture of the basis states of the EA, $\\{\left| j \right\rangle \\}$. Since $| \psi \rangle$ is a mixture of basis states, $|s\rangle \otimes | \psi \rangle$ is also a mixture of basis states (like Eq. (21)). Generally speaking, the underlying state can serve as a classical description of certain aspects of the environment, such as restricting the agent to occupy only one location at each time step, which we considered in our experiments. This representation is crucial for integrating classical and quantum aspects within the framework of EA-MDPs. Based on your comment, we will revise the text to clarify this point and describe the formulation more clearly. We will also add an example to clarify most concepts further.

Response to Q2: We would like to highlight that our model and solution allows the agent to tackle the EA uncertainty-which is encoded via probability amplitudes and superposition of quantum states- by estimating the rewards using an extra expectation over the possible outcomes. In addition, in terms of quantum state observation, we agree with the reviewer and our setting is similar to fully observable MDPs. However, from the perspective of underlying states, our model is not similar to conventional fully observable MDPs as the agent does not observe underlying states directly. Moreover, our model differs from conventional MDPs with respect to reward and value-function definition as there is no outcome set in conventional MDPs; in EA-MDP there is an extra expectation over outcomes which is the result of having probability amplitudes and outcome set. This outcome set helps to assign rewards to the multiple configurations of conflicting evidence. Based on your comment, we add this to the paper to clarify it further for the readers.

We would like to thank you for the precious feedback. Below, we respond to your questions in the same order. We sincerely hope that you find the responses and revisions satisfactory.

Response to Q1: We agree with the reviewer's comment regarding the distinction between ambivalence and uncertainty. That is why, in lines 68-61 of the paper, we first tried to distinguish ambivalence from uncertainty. However, EA can be considered a type of uncertainty, since it represents an agent's uncertainty that arises due to contradictory experiences (please see lines 67-69 of the paper). While traditional uncertainty represents imperfect knowledge, EA represents a cognitive state in which conflicting pieces of evidence coexist, resulting in an uncertain decision making process. We will revise the text and make it more clear based on your comment.

Response to Q2: We included one real-world example in the paper related to stock trading. Another real-world example would be autonomous robot navigation problem where a robot receives conflicting data from multiple sensors; some sensors indicate a clear path ahead, while others detect an obstacle which may be either near or far away. Given this contradictory evidence, the robot is in a state of epistemic ambivalence where it must decide whether to slow down, stop, or maintain its current speed. To simulate a similar scenario, in experiment 2 of Section 6 (Experimental Results and Analysis), we considered multiple site systems, which is a mimicry of the MAZE problem. The goal is to navigate the optimal route to reach the terminal state. In this experiment, multiple EA are applied at different sites and the agent must decide on its next movement at each site.

Response to Q3: Our model is fundamentally different from PO-MDP, as in our model, the reward is generated based on some probability distribution over outcomes. First, while PO-MDP concentrates on situations with partial knowledge, EA-MDP employs quantum-inspired approaches for expressing and managing EA caused by contradicting evidence. Second, the final $\tilde{s}$ is the Kronecker product of $s$ and $\psi$, so we cannot really separate these two components in quantum state $\tilde{s}$ without any extra measurements. Third, in our approach, as shown in proof of Theorem 1, the environment needs to estimate the probability of outcomes and, consequently, estimate the expected reward (8), in order to compute the expectation value of the reward. An environment in a PO-MDP is not aware of such probabilities (is not aware of EA), hence PO-MDP methods fail in our setting.

Dear reviewer KWqS.

We appreciate your time and work when reviewing our paper.

We hope that our replies appropriately addressed your concerns and explained the issues you mentioned.

If there are any remaining questions or require clarification, please let us know. We will be happy to give extra information and discuss them further.

Thank you for your time and for contributing to the review process.

Best Regards,

Authors