Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies

We thank the reviewer for their comments and for the time spent reviewing our paper.

Feasibility of safeguarding black-box policies in SMDPs

We argue that the main issue is not whether safeguarding a black-box policy is feasible, but whether it produces a useful policy. Indeed, the practicality of the safeguarding method comes from the fact that the computational complexity of finding $\mathcal A_{safe}$ can be made as small as desired based on the resources available (with $K$, $C$, $H$, and $n$ being chosen as desired, unlike the PAA policy where they are determined by $\epsilon$, $\delta$, $\gamma$ and $D_{KL}(p\Vert\hat{p})$). However, as these values decrease, the $\alpha$ term in the definition of $\mathcal A_{safe}$ increases, and the resulting policy may become less useful, as only actions with very high Q-values will be considered safe. Such actions might not even exist (empty $\mathcal{A}_{safe}$), in which case the safe policy is essentially useless. On the other hand, deriving the computational complexity to ensure usefulness (i.e., the safe policy always selects an action) cannot be done in the general case, as it would require knowledge of the Q-values for every state-action pairs.

Scale of the reported utilities

We believe it is crucial that assessors are instructed to report their utilities for different states within a unified range, such as between 1 and 10. If this range is not standardized, utilities would need to be artificially mapped to a common scale to be aggregated, which we believe would introduce more distortion compared to when this mapping is done internally by the assessor before reporting. However, even with a unified range, interpersonal comparability is not guaranteed, as any given rating within that range might reflect different levels of welfare across different individuals. This issue is philosophical in nature, and there is no feasible experiment to determine whether utilities are truly comparable.

From this point, two approaches can be taken:

• One can simply treat the utilities as if they were comparable, and construct a social welfare function based on them.
• Alternatively, one can take additional steps to calibrate the utilities using anchor states that should elicit similar welfare across individuals. For instance, in an economic context, assessors could be instructed that a rating of 9 corresponds to landing their dream job, while a rating of 2 corresponds to being unemployed. Although this may not necessarily make utilities more comparable (as people may have different sensibilities to these events), it does provide a more interpretable framework.

In general, while interpersonal comparability might seem like a strong assumption, there are practical examples where it appears evident. For instance, it is generally conceivable that two people can determine who enjoys chocolate more through a conversation. This necessarily implies that their utilities have a certain level of comparability.

We thank the reviewer for their comments and for the time spent reviewing our paper.

Usefulness of the theoretical results

We understand that the reviewer's primary concern is the lack of clarity regarding the applicability of the theoretical results to real-world scenarios. While we do not expect PAA (or safe) policies to be implemented in practical applications in the immediate future, we believe this work lays a foundation for future research, where relaxing some constraints could make the approach more practical. For instance, as mentioned in our response to review Fxrx, one potential relaxation is using surrogate models to provide the necessary feedback for constructing the alignment metric. Another relaxation to address the issue of world model accuracy is to develop world models biased towards predicting catastrophic state-action pairs, as errors in predicting non-critical outcomes are less problematic. The rationale behind the paper's fully theoretical approach is to emphasize the existence of a priori alignement (through PAA policies) rather than presenting and testing a specific policy, with the hope that it will spark further research in this direction.

Practicality of the proposed PAA policy

Regarding the practicality of the PAA policy introduced in the paper, it is designed primarily to facilitate mathematical proof of its PAA property in a general setting (hence demonstrating the existence of PAA policies in any context). The policy essentially employs a brute force approach by testing all possible actions to identify the best one. However, in a real-world scenario, domain expertise and other simplifying assumptions would certainly be leveraged to eliminate certain state-action pairs, significantly reducing the algorithm's complexity. We intentionally omitted an experimental section to avoid giving the impression that one of the main contribution of the paper is the PAA policy itself, and that it should be implemented as is.

Are we simply shifting the difficulty to building better world models?

The main question of the reviewer is a very important one. We address it by exploring two key implications of our work:

• Firstly, as mentioned in the paper’s introduction, a priori alignment is feasible only if we have a sufficient understanding of the consequences of each action available to the autonomous agent. However, our results offer an alternative perspective: they indicate that critical actions should not be delegated to an autonomous agent if the available world model is poor when it comes to predicting their effects. In that sense, our work is the first to provide a quantitative framework for determining which actions can be safely entrusted to an autonomous agent. We believe this type of analysis alone could initiate several research threads in the near future.
• Secondly, while it is true that creating a statistically accurate world model can indeed be as challenging as developing an aligned AI agent, achieving consensus on what constitutes a good world model (i.e., one with high empirical accuracy) is generally more straightforward than defining what constitutes an aligned AI agent (i.e., one with sufficient alignment, which is typically poorly defined). In other words, while our work may shift the challenge to another complex problem, it is one where the solution is easier to verify and where the objective is better defined.

We thank the reviewer for their comments and for the time spent reviewing our paper.

We understand that that the primary concern of the reviewer is the gap between the theoretical result presented in the paper and their practical implementation in real-world scenarios (in particular for safe policies). To address this concern, please find below a detailed discussion about the implementability of these policies in several applications, along with a general roadmap.

• Healthcare decision making: We consider a personalized autonomous doctor (black-box AI agent) for a specific patient. Each week, based on the patient’s health data, the agent determines the appropriate medication for the patient by consulting the patient about various alternatives about aspects such as tolerance to side effects, budget constraints, and preferred timing of administration. In that scenario, the world model could simply be the rate of success and of side effects associated with each medication/dosage during the trial phase of that treatment. Based on these factors, the patient rates each medication alternative on a scale from 1 to 10, and can decide on a minimum discounted welfare level $\omega$. If the agent cannot find a verifiably safe action among its proposed alternatives, it halts and refers the patient to a human doctor. Based on the given guarantees, there will still exist (with high probability) a medication path that generate a future discounted welfare of at lease $\omega$. A certified doctor (or several) could also be directly involved in the process. This involvement could take two forms: (i) the doctor acts as a stakeholder accountable for the AI's recommendation, providing feedback on the risks associated with each alternative (also on a scale from 1 to 10 , which is then aggregated with the patient’s score to form a global welfare score), or (ii) the doctor serves as a world model, predicting the impact of each alternative on the patient's health in the absence of other predictive models.

• Financial trading: We consider the example of an autonomous mutual fund, where investors pool their money to generate returns. The investors could collectively set an aggregate tolerance threshold, $\omega$, that the autonomous trading system must guarantee. Before executing any trade, the system could present various trading strategies along with their associated risks to a subset of the investors. These investors would then assign a utility score (e.g., from 1 to 10) to each option, considering factors like risk and investment type. If no strategy meets the risk tolerance $\omega$, the autonomous trader would abort the transaction and either defer to a human trader or return the funds to the investors. This mechanism ensures that each investor can trust the autonomous agent with their capital, offering a level of verifiable security not always present with human traders. This type of framework could strongly appeal to ethical investors, who prioritize not only the financial returns of their investments but also their moral implications.

• Autonomous driving: In such a scenario, where decisions need to be made rapidly (likely at the sensor acquisition rate), direct human feedback is typically not feasible. However, cars could communicate with each other. For instance, when a car intends to change its trajectory (such as overtaking, turning, stopping, or accelerating), it could broadcast various proposed paths to nearby vehicles. These neighboring cars would then assess the risk (e.g., collision potential) of each trajectory based on their own planned actions and surroundings (which might not be visible to the initial car). The initial car would then select a safe trajectory or abort the maneuver and return control to the driver if no safe options are available. Although this may not be the most natural application of PAA policies, we believe that designers of autonomous vehicles could benefit from approaching the autonomous driving problem through a social choice framework, where each car would act as a stakeholder within the environment (i.e., the roads).

• General roadmap: In general, to construct a safe policy from a black-box policy, several factors must be considered. First, we need to establish acceptable tolerance levels, $\omega$ and $\delta$. Next, we must select values for $H, K, C, n$ based on the available resource constraints, such as the number of feedback instances we can gather and the frequency with which we can query the world model to plan each action. Increasing these hyperparameters (excluding $\delta$) typically leads to a lower $\alpha$, resulting in a safe policy that is less prone to delegating its decision authority (i.e., that is more useful). The accuracy of the world model also influences $\alpha$, with a higher accuracy producing a more useful safe policy. If the resulting safe policy is not useful enough, it suggests that either the resources allocated are inadequate or the world model lacks sufficient accuracy (i.e., $\alpha$ is too large). In both cases, it indicates that we are not yet ready to safely use the black-box policy for that particular application. However, if the amount of feedback required is the bottleneck, one can also imagine a proxy setting where each stakeholder delegates it’s ability to provide feedback to a personalized model (similar to a reward model) that can automatically provide feedback on a given state without human intervention. Note that the safety guarantees would be valid on the alignement metric computed with these surrogate models rather than with the real utilities.

We thank the reviewer for their comments and for the time spent reviewing our paper.

It is primarily mentioned that the problem is not well presented. We agree that clarity in presentation is crucial and would greatly appreciate specific suggestions on how to improve it. Below is a detailed discussion about the two specific examples that are put forward:

• Equation (7): To prove our main contribution, the existence of PAA policies (Theorem 2), we construct such a policy in Section 3.2. This policy is built on a near-optimal planning algorithm that approximates Q-values using a world model. Once these Q-values are obtained, the policy selects actions with the highest Q-value, similar to many common reinforcement learning algorithms. Equation (7) formally defines this policy. The technical core of the paper demonstrates that this policy is indeed PAA. In Section 3.2 (with full proofs in the Appendix), we show that the approximate Q-values can be made arbitrarily close to the optimal Q-values. While this specific policy is primarily designed to be provably PAA across various applications, we acknowledge that is is one of many possible PAA policies and we anticipate that future work may develop more efficient algorithms for specific applications.
• Equation (3): This is still a prior approach according to our definitions because the future states $s'$ are sampled based on the world model $\hat{p}$ rather than the true environment $p$. Thus, the agent does not need to perform actions in the real world to learn the optimal actions. Instead, it can plan according to $\hat{p}$ to ensure that its next action is quasi-optimal with high probability. The policy is therefore a priori aligned, as it ensures alignment before any real actions are taken.

While the main concerns of the reviewer focus on presentation and clarity, we would greatly appreciate a more detailed feedback related to the paper's soundness and contribution, which would help us address the concerns leading to the borderline score. In particular, we would like to understand if the reviewer found that some claims could be better supported (soundness), and if these claims lack originality (or significance) with respect to existing literature (contribution).