A Multi-Agent Reinforcement Learning Framework for Evaluating the U.S. ‘Ending the HIV Epidemic’ initiative

Thank you for the insightful questions.

Yes, that is correct. In the single-agent (now aggregated-SARL or A-SARL), one aggregated national-level action is selected and applied to all jurisdictions. In the multi-agent model (MARL), each agent is evaluating an action specific to its own jurisdiction, which makes it decentralized. We had a third scenario that was presented in the appendix, which we now refer to as independent-SARL or I-SARL, which represents evaluating each jurisdiction independently, i.e., ignoring the epidemic interactions across jurisdictions. A-SARL and I-SARL are representative of models that are used in current HIV literature. We have rewritten the introduction and modified several parts of the paper to more clearly discuss this.

Yes. The agents are acting in an environment with epidemic interactions across jurisdictions, i.e., sexual partnerships can form between persons of different jurisdictions. Thus, the actions taken in one jurisdiction will affect the infection rate of its neighboring jurisdictions, i.e., the state transitions and rewards of each jurisdiction are influenced by the actions taken by all jurisdictions. We made modifications to the methods to clarify these points. 3. We assume it is fully cooperative as in reality. In the model, while the current state is known to all agents and agents share a common objective, as each agent evaluates its own action and all agents simultaneously evaluate their actions, the evaluated state-transition will be different than actual state transition. Therefore, from an agent’s perspective, the environment presents properties of non-stationarity, making the formulation equivalent to Dec-POMDP in model-based methods. For purposes of scalability or in the model-free context as ours, MARL methods have been evaluated as approximate solution algorithms for Dec-POMDP [Hyun-Rok et. al. Operations Research, 2021; Shayegan, et al., PMLR, 2017]. In this work, we focused on evaluating the significance of MARL instead of SARL as in current HV modeling literature. In future work, in application of the model to the U.S, we plan on evaluating multiple solution algorithms suitable for this context. 4. That is a great point that, generally, this application would require a non-stationary policy if modeling epidemic prevalence as the state, and including time-step ‘t’ into the state space can do that. In our case, though we did not prove this and is a limitation, the chance that the same state occurs at two different time-points is low, because of a few points related to our formulation and the application features as follows. Our state is formulated as a conjunction of prevalence (proportion infected) and care metrics (proportion aware, proportion VLS, and proportion PrEP). We set constraints on care metrics to be non-decreasing over-time. HIV has no cure, and therefore unless deaths exceed incidence, prevalence is non-decreasing. In the case that deaths are lower than incidence, i.e., prevalence is non-decreasing over time: If care metrics increases at a slower rate, prevalence increases at a faster rate, and thus, having the same care metrics at two time-points would have two difference prevalence, and similarly, having same prevalence at two time-points can only be generated from two different care metrics. In the case that deaths are lower than incidence, prevalence can decrease over time. However, as people with HIV can live almost normal lives, deaths can exceed incidence only when incidence is very low, which can happen only if care metrics are very high. Therefore, as before, the same prevalence cannot be generated by two different care metrics and vice-versa. However, without a formal proof, there is a chance that we may overlook certain scenarios that may lead to same state at two different time-points (a limitation of our work), but we believe the chance of those scenarios are small. Adding a ‘t’ would increase complexity of training, so our current formulation is more desirable. Though adding care metrics into the state formulation also increased the complexity, it has several other useful properties (as shown in previous work [1]). However, with the availability of more time, we hope to conduct numerical analyses that include ‘t’ in the state-space to evaluate its significance in our current application.

Although every jurisdiction has different sources of funding, public and private, in economic modeling terms, we use a ‘societal perspective’, as is the typical case when evaluating national policies. Specifically, the national policy sets the guidelines, then the necessary resources are allocated through multiple funding sources. Here we assumed a fixed national budget, and evaluated how to optimally allocate it across jurisdictions. In application of the model to inform the EHE (Ending the HIV Epidemic), we will evaluate optimal policies under different budgets, to collective identify resource needs and optimal policy.

The environments are similar in both SARL (now A-SARL) and MARL. The difference is that MARL identifies an optimal policy for ‘each’ jurisdiction while A-SARL identifies an ‘aggregated’ national optimal policy. Kindly note that we compared MARL with two different models (now A-SARL and I-SARL), each representing the two broad categories in current HIV modeling literature. Kindly note that the scope of this paper was to demonstrate the use of MARL to overcome current gaps in the HIV modeling literature. We have rewritten the introduction and modified parts of the paper to clarify this.

We agree that it is extremely important to have validation steps. Thus far we have taken precautions to extensively validate the environment (in prior work Tatapudi et. al., 2022]) and in data inputs. The data inputs are from well-established CDC’s survey and surveillance systems, with detailed cost functions informed through the literature. However, the data for the cost functions are limited when related to outreach and support programs, or as new technologies evolve (e.g., higher efficacy in HIV self-tests, which may have more uptake than clinic-based tests). Therefore, in future work, when applying this model to inform policy, we plan to do extensive sensitivity analyses. We also additionally plan to compare our model results with results from status-quo policy, under the range of uncertainty assumptions for all the sensitive input values, and under varying cost functions. This will help to evaluate if our model’s optimal policy would outperform the status-quo policy in all input assumptions or only in certain cases, which will help make informed decisions.

The status quo policy acts as the baseline policy in our problem.

Thank you for the insightful review.

1. Thank you very much for raising this concern. We understand that the paper might be difficult to comprehend for the readers outside of the HIV and public health domain. We have added a terminology definition in the Appendix of the paper.

2. The reviewer is correct that the main difference between MARL and SARL is that MARL identifies an optimal policy tailored to a jurisdiction while SARL identifies an aggregated national optimal policy. We now refer to SARL as aggregated-SARL ot A-SARL. Kindly note that we compared MARL with two different models (now referred to as A-SARL and Independent-SARL or I-SARL), each representing the two broad categories in current HIV modeling literature. To note, current models in the literature are either national i.e., evaluate one aggregated policy for the nation, or independent jurisdictional, i.e., do not model the mixing across jurisdictions but estimate actions specific to that jurisdiction. We formulated A-SARL to represent the former. To represent models of the latter type, we modeled I-SARL, where we evaluated one jurisdiction independently, i.e., evaluated optimal policy under assumption of no cross-jurisdictional interactions (no-partnership-mixing). Figure A.1 (a) compares results between MARL and I-SARL. Results suggest that discarding the effect of mixing leads to different policies with different impacts. Without loss of generality, we can conclude that if we do this analyses for each of the N jurisdictions, i.e., estimate optimal policy for each of the N independent jurisdictions, the resulting optimal policy will differ from that obtained from MARL. We believe the reviewer’s suggestion on having a centralized agent that output N actions can be considered representative of this type of analyses. We have rewritten the introduction and modified several parts of the paper to more clearly discuss this.

This is a great suggestion. We are trying to use a similar method for the extension of our work over all the jurisdiction in the US. In this paper, our aim was to evaluate the advantages of multi-agent approach compared to single agent, and thus, we used a simple case of multi-agent reinforcement learning.

We used multiple seeds in our training to show the convergence of the algorithm. Due to page limitations, we did not include those results from various random seeds. We will add a short description of this in the Appendix.

Though there are interactions across agents we assume a non-competitive and cooperative environment, as is the case in reality. Though Markov games is a broader term that can include these variations, we use the term MDP to avoid implications of competition between jurisdictions.

Thank you for the insightful review.

1. That is correct that for infectious diseases, typically in real-life, proportion unaware are not always know. However, for HIV, the surveillance methods are well advanced that there exists good molecular methods that have been shown to provide accurate estimates. They are routinely estimated as part of CDC’s NHSS (National HIV/AIDS Surveillance Systems) and made publicly available annually. Previous work [Khatami, 2021] showed that this representation of state with the action representation of directly using changes in proportion unaware (instead of testing frequency, which is more implementable) has very good mathematical properties suitable for convergence. First, it removes the infeasible solutions, and second it makes the feasible space convex, thus assisting in faster convergence. Further, once we determine the optimal action (i.e., proportion change in unaware) they can be converted into testing frequency, i.e., given a state, a function mapping the action to testing frequency is a bijection function (one-to-one mapping).

2. That is correct that some infections, such as the COVID-19 pandemic, required constant policy changes to quickly adapt to the changing pandemic given: a) COVID-19 is a fast spreading disease (transmission rate per contact per day ~14%), b) it was a newly emerging infectious disease with no prior knowledge for setting appropriate intervention guidelines, and thus, frequent changes were needed, and c) we believe, it was feasible to implement frequent changes under the declaration of a global emergency. On the contrary, HIV is a slow spreading disease (transmission rate of 4% per year), and thus, the impact of frequent changes would not be feasible to evaluate as part of routine monitoring. The national strategic plans and guidelines are modified less frequently, e.g., the most recent changes in HIV testing guidelines were in 2017 and only for one population (MSM) [2]. Further, we believe, frequent changes in public health guidelines may not be feasible in the case of non-emergency situations given the time it takes to implement changes, such as US congressional appropriations, preparation of healthcare professionals and infrastructure, and changes in insurance policy. However, we chose a shorter timeline of 1 year to provide the flexibility to balance feasibility with need . For example, PrEP guidelines were updated in 2022 because a new drug was FDA approved in 2021. Estimating optimal annual rates provides the flexibility to set appropriate guidelines. For example, with recent advances in HIV take-home self-test kits, there is interest in suitable guidelines for combinations of clinic-based tests v. self-tests (weighted by effectiveness, adherence, etc.) to reach diagnostic rate targets to achieve the EHE (Ending the HIV Epidemic) goal. Our approach of identifying optimal annual diagnostic rate targets fits well in this context.

3. That is correct that in the case of SARL, a single agent learns the actions for all the jurisdictions in it. But in the case of MARL, each jurisdiction (agent) learns its own actions. The interactions across jurisdictions in both MARL and SARL are in the epidemic environment, modeled as sexual partnership mixing across jurisdictions. In the setup of our MARL model, each agent is determining an optimal action based on the current state of the system (current state of each jurisdiction is known to all jurisdictions). However, as state transitions depend on the actions implemented in all jurisdictions (because of sexual partnership mixing), and as all agents (jurisdictions) are simultaneously evaluating this decision, it is infeasible to know the actual state transition. This introduces properties of non-stationarity into state-transitions, making this a suitable application for MARL. We also evaluated a SARL (now called aggregated-SARL or A-SARL) formulation as a representation of current models in HIV literature. Specifically, the models in the HIV literature are typically ‘aggregated’ national or ‘independent’ jurisdictional (no epidemic mixing across jurisdictions). We use A-SARL as a representation of the former type. To represent models of the latter type, we evaluated one jurisdiction independently, i.e., ignored the mixing (presented in Appendix Figure A1). We now refer to this as independent-SARL or I-SARL. The corresponding results differed from that of MARL highlighting that ignoring mixing can lead to non-optimal decisions. The scope of this work was to present the use of MARL to overcome the gaps in current HIV modeling literature. We believe our contribution is developing a comprehensive modeling framework and simulation environment that is suitable for analyses of the U.S. ‘Ending the HIV Epidemic’ national strategic plan. We have rewritten the introduction and modified several parts of the paper to more clearly discuss this.

Thank you for the insightful review. The main difference between the multi-agent model (MARL) and single-agent model (SARL) is that MARL identifies an optimal policy tailored to a jurisdiction while SARL identifies an aggregated optimal policy. The scope of this work was to evaluate the advantage of a MARL type model, as opposed to current modeling approaches in the literature. Specifically, current models in the literature are either national, i.e., do not model jurisdictional heterogeneity but instead evaluate aggregated national-level decisions, or independent jurisdictional, i.e., do not model the interactions (through sexual partnership mixing) across jurisdictions but estimate actions specific to that jurisdiction. We formulated SARL (which we no refer to as aggregated-SARL or A-SARL) to represent the former, for purposes of comparing with our proposed MARL approach. To also note, to represent models of the latter type, we evaluated one jurisdiction independently, i.e., evaluated optimal policy under assumption of no interactions (no partnership-mixing) with other jurisdictions and compared it to results from MARL for that jurisdiction. We now refer to this as independent-SARL or I-SARL. We presented these analyses in the Appendix, in Figure A.1 (a), and the observations are that discarding the effect of mixing leads to different policies with different impacts. The reviewer makes a good point that we could have expanded the action and state space to include a conjunction of all jurisdictions, which would then be reduced to a single-agent model. However, such a model is not scalable. Though in these analyses we used a simple example of 8 jurisdictions, in reality the number of jurisdictions are typically large (96 in our ongoing work applying this model to the national U.S). This creates massive sizes of state and action space. In this example, the size of the action space would be 9^96, which makes convergence challenging. Therefore, we believe, MARL is a suitable approach for this problem To note that, the scope of this work was to demonstrate the advantage of MARL to overcome the gaps in current HIV modeling literature. We believe our contribution is developing a comprehensive modeling framework and simulation environment that is suitable for analyses of the Ending the HIV Epidemic national strategic plan. We have rewritten the introduction and modified several parts of the paper to more clearly discuss this.