Altared Environments: The Role of Normative Infrastructure in AI Alignment

No description on how the altar "dynamic" attribute works. It is not clear to the reader how the altar is set up, i.e. How are punishment policies chosen? Is this hard-coded? This is what is being revealed: "Finally, as the institutional content is dynamic," but there is no indication of how this works. The same goes for the hidden-rules setup. How are hidden rules decided, and what are they exactly? This might become clear when investigating the environment suite; however, it should be made clear in the main body of this paper.

We apologize for the unclear writing in the initial submission that discussed all these questions but in less readable manner. To remedy that, we have not revised the manuscript comprehensively to make all these details very clear, precise and accessible. We first outline the specific revisions to the sections relevant to the concerns raised by the reviewer and then briefly describe them here.

• In Section 2, we introduce the theoretical framework that underpins our methods, provide the formalism of the Sanction Augmented Markov Game (SMG), and discuss how this framework serves as the foundation for the Hidden Rule SMG baseline used in our experiments.
• In Section 3, we address the key challenge of dynamic norms in our setup, offering a formal discussion of this issue. We then present one of our main contributions: the extension of Markov Games to their altered versions, referred to as Altared Games.
• The exact nature and dynamics of the norms for each environment, i.e. how we instantiate the rule sets and how they change over time are discussed in detail in Appendices A.1 and A.2 of the revised version.

Dynamic Norms: For both the hidden rule setup (now called Hidden Rule SMG) and altar setup (now called Altared SMG), the dynamic norm mechanism works in the same way. In fact, all the setup between Hidden Rule SMG and Altared SMG is exactly the same except the presence of institutional features of the environment, called altar in our approach.The controlled nature of our framework ensures that external factors do not interfere with the experimental results, making it possible to isolate and analyze the specific impact of the altar. By holding all other aspects of the system constant, we enable hypothesis testing that focuses solely on the presence or absence of an explicit classification institution. This methodology allows for a rigorous examination of how agents’ learning, coordination, and compliance behaviors change when norms are made observable In our formal setup, the evolution of norms is governed by an update function: $Nt+1 = f (Nt, \phi)$, where $f$ captures the mechanism of norm evolution, and $\phi$ represents triggers or drivers of change. We do not model the determinants of norm evolution but these drivers could be thought of as arising from external inputs (e.g., regulatory updates or environmental changes), agent-driven mechanisms (e.g., collective decision-making or voting), or stochastic events (e.g., resource depletion or unexpected disturbances).

In this work, the norms and corresponding punishment policies are hard code i,e,wWe assume that a normative system is already in place, with predefined but dynamic norms (as discussed in Section 3) and associated enforcement mechanisms. Our focus is on investigating the role of explicit institutional mechanisms, such as the altar, in improving agents' ability to learn and adapt to these evolving norms, particularly in dynamic environments. To the best of our knowledge, the role of institutions in supporting norm compliance for AI agents has not been studied before. Hence, we focus on demonstrating that an ideal normative system produces higher payoffs when agents have access to institutions that enhance their capacity to learn and track the content of norms. This allows agents to adapt more rapidly and reliably to a changing normative structure, irrespective of how those norms originated. While broader questions regarding the emergence and establishment of norms are beyond the scope of this work, readers can refer to relevant literature, such as [1,2] for a detailed discussion.

We will include this discussion in the final version of the paper.

[1] A survey of research on the emergence of norms in multi-agent systems, Bastin Ton Roy Savarimuthu and Stephen Cranefiled, 2011.

[2] Norm emergence in multiagent systems: a viewpoint paper, Andreasa Morris-Martin, Marina De Vos and Julain Padget, AAMAS 2019.

We believe that the updates and responses we have provided address the key concerns raised. We kindly invite the reviewer to revisit their score in light of these clarifications and improvements. If there are any remaining questions or areas where further discussion would be helpful, we would be delighted to engage further and ensure all concerns are fully addressed.

Experiments and Vanilla Baseline

We appreciate the reviewer’s concerns and believe there may have been a misunderstanding regarding the reporting of results, particularly for the vanilla baseline. The vanilla baseline is designed to represent the original environment as closely as possible, retaining its original sanctioning mechanism. In the original setup, sanctioning resulted in a penalty for the target agent but provided no reward or penalty for the sanctioning agent. This is retained in the vanilla baseline. By contrast, both the Hidden Rule SMG and Altared SMG setups include a reward structure for correct sanctioning to incentivize agents to learn sanctioning behavior. The altar-based approach further adds explicit norm representation as a feature of the environment, as discussed earlier.

Regarding fair comparison, we emphasize that we report Adjusted Mean Reward for both the Hidden Rule SMG and Altared SMG setups. This adjustment excludes the artificial positive rewards gained from correct sanctioning, which are designed to train agents, and instead includes:

• Base environmental rewards (e.g., consuming a berry or harvesting an apple).
• The cost of sanctioning.
• The cost of being sanctioned.

This adjustment ensures that the reported rewards reflect meaningful cooperative performance by focusing on sustained outcomes. It demonstrates that the sanctioning mechanism is beneficial only when the environmental rewards outweigh the costs incurred through sanctioning, providing a fair and transparent comparison.This presents even stronger results, as both the Hidden Rule SMG and Altared SMG setups perform better than the vanilla baseline, even when including the cost of sanctioning for the source agent—a feature absent in the vanilla baseline.

Regarding metrics, we agree that metrics such as depletion rate are important in these environments. Sustained high welfare inherently requires agents to coordinate toward socially optimal strategies. For example: in Commons Harvest, agents must avoid overharvesting to prevent depletion, as persistent overharvesting directly reduces overall rewards in subsequent episodes. In Allelopathic Harvest, agents must learn to align with shifting norms to maintain optimal resource use. While Adjusted Mean Reward reflects this coordination, we also report additional metrics directly relevant to norm enforcement, such as rates of correct and incorrect sanctioning. Finally, as shown in Figure 4c, our approach indeed significantly reduces the depletion rate over time. These results indicate that agents not only comply with evolving norms but also enforce them effectively in the presence of institutions.

We hope this clarification addresses the reviewer’s concerns. We welcome further feedback to ensure the metrics and comparisons are clear and meaningful.

No mention of the results of the original authors of the Melting Pot environment suite The baselines are not supporting. The results reported by the original authors of the Melting Pot environment suite are not mentioned, and I believe this work should be compared against what the original work reported.

We respectfully disagree with the reviewer and believe that a comparison with the original results from the Melting Pot environment suite would not be meaningful for the following reasons:

Different Focus and Objectives: Our work does not propose a novel AI algorithm to solve the alignment problem. Instead, we focus on environments that pose normative alignment challenges for groups of AI agents and propose a formal framework for augmenting such environments with classification institutions. The goal is to systematically investigate the impact of explicit institutional mechanisms on agents’ behavior, specifically their ability to engage in enforcement mechanisms and work toward alignment. Thus, our comparisons are between environments (with and without institutions) rather than algorithms. As we also clarify in response to another question and in our revised introduction, our study deliberately uses architecturally simple agents to minimize their cognitive and exploratory burden, making comparisons with complex algorithms, such as those tested in the original Melting Pot work, not applicable.

Modifications to Melting Pot Environments: While we utilize the Melting Pot suite for its high-fidelity implementation of social simulations, we modify the environments to suit the objectives of our study, as detailed in the paper. These modifications ensure the environments are tailored to our investigation of dynamic norms and institutional mechanisms.

Missing Code and Ambiguous Abstract

We apologize for the oversight regarding code accessibility. The code has now been made available at the same link, enabling further investigation of our work.

We have rewritten the abstract to ensure it is precise, self-contained, and accessible to readers unfamiliar with the specific environments.

We thank the reviewer for their thoughtful feedback and for recognizing the paper’s natural flow and clarity. We are especially pleased that the Introduction and Preliminaries effectively frame the broader interest of the work. We have conducted a careful revision to address the concerns raised, which we discuss below.

my biggest question is: What if the agent's reward function is modified, such as the altars policies are set? Adding the altar as an observable sanctioning feature appears to be a complicated dynamic reward that is not given by the environment directly but rather potentially by other agents catching others doing the wrong thing. This would mean that if a negative reward for bad behaviour is always given, we should even see better performance, but this could be achieved by reward-shaping or adding a reward structure already only. In conclusion, the partially observable rules that help agents police each other's actions, when observed, seem not worth the effort.

We are not entirely sure we fully understand the reviewer’s question, so we will first outline our interpretation and then respond accordingly. Based on the comment, we interpret that the reviewer is suggesting replacing the altar and its associated enforcement mechanism with existing approaches, such as reward shaping. Specifically, this would involve directly modifying the agents’ reward functions to penalize bad behavior, thereby eliminating the need for the altar as an explicit mechanism for agents to enforce norms. The reviewer seems to suggest that using the altar introduces unnecessary complexity and is not worth the effort. If this interpretation is incorrect, we welcome the reviewer to clarify their comment further.

Response: Based on this interpretation, the approach proposed by the reviewer is in direct contrast to the objectives of this work. As detailed in the introduction (and clarified further in the revised version), this work aims to shift the focus of AI alignment research from embedding norms and values into AI agents to building normative infrastructure that supports agents in achieving alignment. Approaches like reward shaping (whether using intrinsic or extrinsic rewards) impose a significant cognitive and exploratory burden on the agents themselves, which this work aims to avoid. As we argue in the introduction, humans do not internalize all norms and values; instead, they rely on cooperative interactions with institutions to achieve alignment in groups. This forms the core motivation for our framework and experimental study: investigating the role of institutions in helping AI agents adapt with dynamic norms and stabilize around cooperative outcomes in comparison to environments without institutions.

Regarding the reviewer’s reference to third-party enforcement or “agents catching others doing the wrong thing,” this concept has a long history of study in social anthropology, economics, and law. As noted in the introduction and recognized by Reviewer HNPm, such mechanisms are foundational in human societies and are widely observed in real-world social systems. Our approach builds on this conceptual framework and is grounded in the Hadfield-Weingast model [1], which formalizes the interplay between classification institutions and enforcement mechanisms.

On the comment that the altar introduces a “complicated dynamic reward,” we believe the reviewer is conflating the relation between the altar role and the reward mechanism itself. This is likely due to a lack of clarity in the earlier version, which we apologize for and have addressed in the revision. The key clarification is that the sanctioning reward mechanism remains identical between our baseline (Hidden-Rule SMG) and our approach (Altared SMG). In both cases, norms are dynamic, but in our approach, they are explicitly observable through the institution. The principles governing sanctioning are as follows:

• Sanctioning is costly for both the source (agent who sanctions) and the target (agent being sanctioned).
• Sanctioning becomes rewarding for the source only if the target had violated the prevailing norm in the previous timestep.

In the baseline setup, agents must infer norms and adapt to changing norms solely from the sanctioning reward signal. In our approach, the agents also receive an explicit signal for the prevailing norm via the altar, requiring them to learn to correlate altar observations with the sanctioning reward mechanics. Our results show that agents in the altar setup consistently visit the altar, engage in correct sanctioning behavior more often, and achieve better alignment compared to the baseline.

[1] What is law? A coordination model of the characteristics of legal order, Gillian K Hadfield and Barry R Weingast, Journal of Legal Analysis, 2012.

Missing Citations and Captions

We have addressed the missing citations and revised the figure captions throughout the paper to ensure they are more informative and descriptive.

What the results show, however, is that altar visits decline in both environments while correct sanctioning declines and incorrect sanctioning inclines and as a result, the Adjusted Mean Episode Reward still increases. This means that agents have learned not to go to the altar as much; however, they get worse at sanctioning. On the other hand, the Adjusted Mean Episode Reward still increases. This makes no sense to me. What is contributing to the Adjusted Mean Episode Reward if the correct sanctioning is declining? It cannot come from correctly sanctioning. In order to answer this question, we are missing results.

The concerns here again stem from a misunderstanding about our core metric, Collective Return (originally called Adjusted Mean Episode Reward). What contributes to Collective Return is higher consumption. These results are not missing; they are the core metric we report.

The reviewer also has misunderstood the altar visit metric and they have misread the sanctioning plots. Let us clarify the following:

• Altar visits are reported in Figure 4 for both the environments. We request the reviewers to check the Y-axis . Initially agents are visiting the altar at a significantly high rate. It is expected that this rate will drop significantly as the agents learn the environment. Visiting an altar has an opportunity cost – during this time, agents could have instead consumed berries or apples. As discussed in the details in Appendix A.1 and A.2, the norms do not change very frequently, hence it does not make sense for agents to keep visiting the altar and over time. They stabilize around number of visits which is around 30 for Allelopathic harvest and around 500 in Commons Harvest (the high number in Commons Harvest is because there are three altars and smaller map so easier to visit them repeatedly, less opportunity cost).
• Correct Sanctioning for our approach (Altared SMG) always increases and improves over the baseline Hidden Rule SMG (see Figure 2b) for Commons Harvest environment. Incorrect sanctioning for our approach (Altared SMG) always decreases and stays below the baseline Hidden Rule SMG (see Figure 2c).
• For Allelopathic Harvest, Correct Sanctioning slightly decreases for Altared SMG and incorrect sanctioning slightly increases in the center of training. This is plausible because free-riding (grey agents) is considered a violation of norm throughout the episode and hence initially there are many free-riding agents, and sanctioning them is a correct action. But as soon as agents learn not to freeride, the number of correct sanctions coming from that source declines. In any case, we again refer the reviewer to look at the y-axis of Figure 3b and Figure 3c. The overall correct sanctions still remain above 60% and the sanctioning behavior of Altared framework is improved over the Hidden Rule SMG towards the end.
• Again, to emphasize, Collective Return (formerly Adjusted Mean Reward) should not be increasing or high just because of correct sanctioning, as sanctioning reward is not included in that number. It is indeed high because agents are learning to improve on the base environment rewards.

We hope these responses help to allay the reviewer’s concerns.We welcome the reviewer to raise further clarification questions if there still remains any confusion and we would be happy to help clarify them.

With the results and experiments, you are showing that your mechanism helps agents align on dynamic norms, but what for?

We admit to being very confused by this question as it suggests the reviewer has not understood the most basic goals of our paper. If agents align on dynamic norms, the ‘what for’ is: to better solve the fundamental social dilemma posed in the substrate (Commons Harvest or Allelopathy). We selected the norms to be ones that improve Collective Return by reducing depletion of a limited resource or by coordinating agents on a monoculture and limiting free-riding. It of course follows directly that if we can show improved norm-following (because of improved correct sanctioning in accordance with the norms), then we show improved performance (which we measure.) Our hypothesis is that norm-following is improved with an altar because the altar improves sanctioning when norms are changing (because the best solution to the social dilemma changes) and this is exactly what our results show. That’s what all of this is “for”.

There are no fine-grained results / detailed results, that means potentially the only thing agents get right is sanctioning while they completely fail to complete the tasks / objectives part of the environment.

Again, this comment is very worrisome to us and we hope it is only because of the core misunderstanding about what is measured by what we called Adjusted Mean Episode Reward in the paper and now call Collective Return. It is absolutely not the case that all our agents get right is sanctioning–what they get right because of correct sanctioning is planting or consumption behavior. They plant more and plant the correct color more successfully in Allelopathy and they avoid consuming apples from patches at critical risk of depletion in Commons Harvest. We present the fine-grained results about Collective Return in both cases which show the impact of these behaviors. We report depletion in Commons Harvest. We admittedly do not show the monoculture fraction in the main paper–as this is very hard to measure in the dynamic setting and we are still working on it–but we report this metric as a ‘sanity check’ in the Appendix D for the static norm case. We will also include videos that show the detailed behaviors in the final version. We’re not sure where this comment is coming from, other than a misunderstanding.

We thank the reviewer for their prompt response for summarizing their understanding of the paper. We’re afraid that the reviewer's comments display several misunderstandings of our approach and setup, which seems to have been exacerbated even further by misreading of our overall results and plots. To remedy that, we will first describe our approach briefly here and then respond to specific comments about our results.

A quick summary of what I understand from the paper:

• Agents go to the altar to check on the latest sanctioning norms (infer norms).
• If agents observe other agents doing something sanctionable, according to the altar, they can sanction them and get a positive reward for correctly sanctioning.
• If agents sanction others incorrectly, they get a negative reward.

While the above summary is correct, it is incomplete and so may be the source of misunderstanding. Agents do learn to visit the altar (that’s an important and interesting result in and of itself) and they do get positive rewards for correct sanctioning and negative if incorrectly. But the reason that correct sanctioning produces group payoffs is that it aligns the entire group on a better collective behaviors: all agents plant the same color berry in Allelopathy and all agents avoid harvesting from depleted apple patches in Common Harvest. Because the entire group is coordinated on socially-beneficial behaviors, individuals can consume more and this is the main source of reward in the environment. The average reward metric is primarily based on the level of consumption agents achieved, adjusted for the costs incurred from sanctioning and being sanctioned. We do not include the rewards for correct sanctioning as sanctioning is group-beneficial only to the extent that it generates higher consumption by helping to solve the social dilemma of over-harvesting or uncoordinated planting. Note also that if sanctioning is reliable, agents learn to predict this and avoid it: by complying with the current norm. In a successful group, there will be high rates of compliance and hence low rates of actual sanctioning.

This comment helped us to recognize that our use of the term “rewards” in our principal metric of Adjusted Mean Rewards was confusing. Although we clearly defined the metric on line 462 in the paper and so its meaning can be seen in the math, we have changed the name of the metric to Collective Return, the term used in the original papers (and in Koster et al 2022, on which we build). This focuses on the core behaviors of interest from a welfare point of view: benefits for agents generated from consumption and costs associated with sanctioning. Again, rewards for correct sanctioning do not enter into the Collective Return (formerly Adjusted Mean Episode Rewards) metric.

I don't understand why we still would not be interested in metrics/objectives that are part of the original environment. The sanctioning mechanism aims to increase performance on such metrics / objectives.

We ARE interested, and report on, metrics and objectives from the original environment from Perolat et al (2017) and Koster et al (2020). Note that agents in both papers also have sanctioning mechanisms (zapping beam); we are not testing the impact of sanctioning per se on these metrics. The most important metrics are those that measure how successful the agents are at solving the social dilemmas: increasing the apples or berries they can consume. This is what our Adjusted Mean Episode Reward Metric (now renamed Collective Return, as we mentioned above) captures in both environments. Perolat calls this “Efficiency”, Koster (2022) calls this “Collective Return”. We also report apple depletion (a metric in Perolat) and are still working to report monoculture (this is much harder to measure in our dynamic setting; our appendix shows a monoculture metric for the static case which is more directly comparable to Koster 2020.) The measures in our environment and theirs are not directly comparable.

Notably, their sanctioning mechanisms are costless to use–ours are costly. Ours are incentivized both by private benefits (as in Perolat and Koster) and by social benefits (rewards for correct sanctioning). Most importantly, however, we want to emphasize that the goal of our work is not to develop an algorithm that outperforms those in these original papers; so we are not measuring our performance against those benchmarks. We are testing the hypothesis, grounded in Hadfield & Weingast (2012), that an institutional feature improves performance in the sense of improving Collective Return. That test is conducted by comparing our Hidden Rule metrics to our Altar metrics. Our results support the hypothesis and generate the important result that improving performance on a social dilemma for multiagent systems can rest not ONLY in improving algorithms, but also in improving environments.

Thank you for your feedback.

By fine-grained results I'm asking for the bare minimum: Metrics that support your claims and beyond from the metrics avialable:

Environment-based (Allelopahtic Harvest):

• ripe berries by type
• unripe berries by type
• berries by type
• coloring by player
• eating types by player
• berries per type by color of colorer
• berries per type by taste of colorer
• player timeout count
• color by color zap counts
• color by taste zap counts
• taste by taste zap counts
• taste by color zap counts
• who zapped who

Algorithm based (RLlib PPO - used by you for training):

• policy loss
• value function loss
• total loss
• entropy
• Kullback-Leibler (KL) divergence
• learning rate
• clip fraction
• approximate Kullback-Leibler (approx_kl)
• explained variance
• timesteps per second
• rewards
• ...

And finally, custom metrics that you might have been tracking to support your claims in regards to your mechanism. As you have provided:

• Mean Episode Reward
• Adjusted Mean Episode Reward (This is still named incorrectly, against your claims of having this changed, in your latest revision in Figure 2, adding to continued confusion).
• Correct Sanctions Fraction
• Incorrect Sanctions Fraction
• ...

I don't think I'm asking too much for authors to provide full result metrics on their experiments.

I have tried to go deeper into your code, but there is no README.md provided to explain what exactly has been done to reproduce your results. Furthermore, I have realized that you were right. I do not fully understand the background of this work, but it is incredibly hard to understand all the implementation details of your mechanisms. I will adjust my rating to reflect this.

We are thrilled to hear that the reveiwer finds our overall approach very good, intuitive, and well-grounded. Your thoughtful and detailed review, particularly your extended efforts in identifying presentation flaws, is greatly appreciated. We sincerely apologize for the shortcomings in our initial writing. Your feedback has been invaluable and has motivated us to significantly improve the presentation of our work, elevating it by an order of magnitude. Below we address your specific concerns:

Issues with clarity, flow and presentation of Section 2, Section 3 and Section 4

As mentioned in our general comment, we have revamped these three sections in response to your valuable feedback. We have streamlined them with a strong emphasis on clarity, accessibility, and accuracy, ensuring they are significantly improved and better aligned with the goals of the paper.

Specifically, we have worked to ensure consistent definitions and references across all sections, including the experiments and corresponding plots.

• In Section 2, we introduce the theoretical framework that underpins our methods, provide the formalism of the Sanction Augmented Markov Game (SMG), and discuss how this framework serves as the foundation for the Hidden Rule SMG baseline used in our experiments.
• In Section 3, we address the key challenge of dynamic norms in our setup, offering a formal discussion of this issue. We then present one of our main contributions: the extension of Markov Games to their altered versions, referred to as Altared Games. We do not use the term I-SMG anymore in our paper and also rectified any inconsistencies and redundancies around its description. Throughout these sections, we have ensured consistent referencing of all methods.
• We have provided a detailed explanation of the core mechanics of the environments in Section 4.1. Additionally, we discuss the three experimental conditions tested in our work in Appendices A.1 and A.2. As noted earlier, we would ideally like to bring Appendix A to the main paper but for now, we have updated the paper for full clarity on our proposed approach, related background and empirical investigation. We will reorganize further to get it in main paper in the final version.

Improvements to Figures:

We acknowledge that Figure 1 is not very informative in the earlier draft. We have updated its caption to better explain our framework and are in the process of improving the figure itself, which will be included in the final version of the paper. We have also revised the figure captions throughout the paper to ensure they are more informative and descriptive. Finally, we have corrected the plot colors and fixed figure references throughout the paper to ensure visual and contextual consistency.

No ethics section? Who gets to decide the norms that the institution classifies as “right”?

Thank you for raising this concern. To answer your question, norms in multi-agent systems can originate in several ways. They may be exogenously provided, pre-defined by system designers or external entities, and encoded into the environment to represent ideal behaviors. Alternatively, norms can emerge from interactions among agents, evolving organically through repeated interactions and collective decision-making processes. Both approaches have been extensively explored in fields such as sociology, political theory, and artificial intelligence. However, this study does not aim to explore the origin or establishment of norms, nor does it investigate how rules should be designed or who determines their content.

We assume that a normative system is already in place, with predefined but dynamic norms (as discussed in Section 3) and associated enforcement mechanisms. Our focus is on investigating the role of explicit institutional mechanisms, such as the altar, in improving agents' ability to learn and adapt to these evolving norms, particularly in dynamic environments. To the best of our knowledge, the role of institutions in supporting norm compliance for AI agents has not been studied before. Hence, we focus on demonstrating that an ideal normative system produces higher payoffs when agents have access to institutions that enhance their capacity to learn and track the content of norms. This allows agents to adapt more rapidly and reliably to a changing normative structure, irrespective of how those norms originated. While broader questions regarding the emergence and establishment of norms are beyond the scope of this work, readers can refer to relevant literature, such as [1,2] for a detailed discussion.

We will include this discussion in the final version of the paper.

[1] A survey of research on the emergence of norms in multi-agent systems, Bastin Ton Roy Savarimuthu and Stephen Cranefiled, 2011.

[2] Norm emergence in multiagent systems: a viewpoint paper, Andreasa Morris-Martin, Marina De Vos and Julain Padget, AAMAS 2019.

We thank the reviewer for recognizing the novelty and strength of our results. Thank you for pointing out the presentation issues which have certainly helped to improve the quality of the paper and we have fixed all of them in the revised version of the paper.

The contribution of this work is can be boiled down to adding manually programmed indications of social rules explicitly into the agents' observations. This is done with no theoretical justification and relatively weak intuition. You should either mathematically prove that using the altar modification encourages correct sanctioning, or at least give a stronger intuition as to why this effect might occur.

We respectfully disagree with the reviewer’s comments on a fundamental level.

The key objective of this work is to draw inspiration from models of human societies to investigate whether a normative institution can improve the capacity of architecturally simple AI agents to adapt to dynamic norms while interacting in mixed motive environments. We are ultimately interested in demonstrating that the human challenge of aligning AI agents with our preferred outcomes could be better solved by expanding our efforts beyond algorithm design to environment design. To conduct this investigation, we systematically design our approach, each of which represents a preliminary but foundational step towards these broader research questions on building scalable, adaptable, and socially aligned AI systems.

First, we ground our formal approach in the theoretical framework of the rational agent model of normative social order. We adopt this framework in the context of AI research, extending Markov games to include an explicit environmental feature called the altar—representing an authoritative focal point that articulates a group’s shared norms or laws. We then tackle the challenging setup of dynamic norms, providing both a recipe and an example to convert a given Markov game into its Altared version. To empirically investigate our framework, we employ a methodology of controlled hypothesis testing, inspired by social science, to isolate the role of the altar. This approach ensures that the effects of the altar can be assessed independently by holding all other factors constant across experimental setups. The manually programmed indications of social rules, as mentioned by the reviewer, are a deliberate design choice to enable this controlled testing and not a weakness of the work (see our detailed response to reviewer BYmR). As reviewer HNPm correctly pointed out, our formal approach and experimental design are deeply rooted in theoretical frameworks from social science and anthropology, supported by strong real-world intuitions and examples from human societies. Finally, we demonstrate that explicit classification institutions, such as the altar, improve agents’ ability to cooperate, enforce norms accurately, and adapt effectively to uncertainty.

The authors claim to have published their code for reproducibility, but following the link provided in the abstract leads to an empty repository

We apologize for the oversight regarding code accessibility. The code has now been made available at the same link, enabling further investigation of our work.

why is POMG mentioned? are the environments partially observable? if so, this is not clear from the text.

Yes, both the environments are partially observable and the agents only have access to their cloak observation. We have described the details on the observations in Appendix E.

In figure 6 the authors show 6 plots. why does figure 7, which is essentially the same kind of results but for a different environment, present only 3?

Please refer to Figure 2 and Figure 3 in the revised manuscript. These figures now present the results in a concise and precise manner. Previously, Figure 6 included two rows, but the second row did not display meaningful data. Specifically, the bottom row in the earlier version reported raw Mean Episode Rewards. However, as clarified in the updated manuscript, we should be looking at the Adjusted Mean Reward for both the Hidden Rule SMG and Altared SMG setups. This adjustment excludes the artificial positive rewards gained from correct sanctioning, which are designed to train agents, and instead includes:

• Base environmental rewards (e.g., consuming a berry or harvesting an apple).
• The cost of sanctioning.
• The cost of being sanctioned.

This adjustment ensures that the reported rewards reflect meaningful cooperative performance by focusing on sustained outcomes. It demonstrates that the sanctioning mechanism is beneficial only when the environmental rewards outweigh the costs incurred through sanctioning, providing a fair and transparent comparison.

We thank the reviewer for recognizing the strength of our experiments and that they show solid evidence of the hypothesis we sought to test, namely that the introduction of a normative institution–the ‘altar’ which represents the state of a dynamic ‘rule’ for the agents–improves the average reward achieved by the agents. We note that the reviewer’s description of the strengths of the paper fully captures our goal and contribution, although the summary of the paper is fairly confusing and may mislead someone who has not read the paper.

We are encouraged that the reviewer says they are open to being persuaded to adjust their score because the weaknesses that concern the reviewer stem from a misunderstanding of our aims and the nature of our contribution. The reviewer appears to believe that our primary goal is to demonstrate improved performance in the Commons Harvest and Allelopathic Harvest environments through tagging (punishing) according to rules. Given this mistaken belief, the reviewer is appropriately concerned about where the rules come from. However, this is not the objective of our work. As prior literature (e.g., Vinitsky et al., Koster et al.) has already shown, MARL agents can learn to punish according to predefined rules, and when these rules incentivize cooperation—such as coordinating to plant same color in Allelopathy or reducing over-harvesting in Commons Harvest—then this punishment improves performance.

Our focus, instead, is on demonstrating that given the right rule and a reward structure incentivizing agents to punish violations of that rule, agents are able to improve their ability to learn and implement the rule–achieve better performance in the face of a social dilemma– when the environment includes an institutional feature (the altar) representing the rule, compared to when it does not. This is why we compare two specific conditions: (1) the baseline hidden rule setup (Hidden Rule SMG), and (2) the altar-augmented setup (Altared SMG). The reward structure for punishment, which is controlled by the experimenter, is identical across both conditions, ensuring that the comparison isolates the effect of the altar.

Hence we are testing only the impact of the presence of a feature that represents the rule. We are not testing the impact of the content of the rule but rather controlling it to conduct a valid experimental test of our hypothesis: that the presence of the altar feature itself causes improved performance. Our approach aligns with social science methodologies, which emphasize controlling all other sources of variability to isolate the causal impact of the variable of interest. This design allows us to rigorously test the hypothesis that explicit institutional representation enhances agents' ability to learn and enforce norms, ultimately improving outcomes in social dilemmas. (This is an experiment to test the impact of this environmental feature, not a more conventional experiment to show improved performance from a new algorithm compared to benchmark performance.) Our approach aligns with social science methodologies, which emphasize controlling all other sources of variability to isolate the causal impact of the variable of interest.

We think this experimental design has been misunderstood in our paper and we have revised our exposition to help ensure readers do not make this misinterpretation of our work. We now address your specific questions in order:

Does the introduction of the altar not add a new observation data point into the input? Or is this just directly learned over time and from passing it in earlier states? I am assuming the latter?

Yes, the altar adds a new observation and this is precisely what we show: with this informational enrichment of the environment, agents are able to better learn the structure of punishment rewards. In the baseline condition, agents have to learn what punishment actions earn a reward solely from observing the environment and tracking how their actions produce rewards. In the altar condition, they have all this information plus an additional observation: a color on the altar. What we are showing is that adding this observation makes the learning problem easier and, moreover, that the agents learn to visit the altar in order to obtain this observation. We make the content of the altar dynamic so as to demonstrate this effect.

Updates to Experiments

• We have streamlined the explanation of the basic environments Section 4.1 and the process of converting them to their Altared versions in Appendix A.1 and A.2. To achieve this, we have relocated the details of the agent architecture to the Appendix and expanded the description of each environment. Each subsection is now self-contained, providing a clearer and more comprehensive overview for each environment. We would also like to bring details from Appendix A to the main paper but will need either an extra page or further reorganization.

• We have addressed concerns regarding variance in the plots by running all experiments with additional seeds and extended training durations. For Commons Harvest, our results continue to demonstrate strong performance compared to the baseline without institutions. For Allelopathic Harvest, we observe improved performance, and, more importantly, reduced variance in our approach over training time. This reduction in variance indicates that agents adapt to dynamic norms more effectively with the inclusion of institutions in our framework.

• We have refreshed all plots to enhance their visual quality and accessibility. Additionally, we have streamlined the discussion of qualitative results to improve clarity and coherence.

• The missing code, as noted by a couple of reviewers, was an oversight on our part. It has now been made available. It is available at the same link: https://anonymous.4open.science/r/normarl-public-804C and the implementation of environment conditions can be viewed in the meltingpot/configs/substrates folder.

Beyond specific concerns, we emphasize that our work makes several fundamental contributions:

• Formally extends Markov games to incorporate a classification institution to study the impact of explicit normative infrastructure on alignment.
• Provides a generalizable recipe and examples for converting a given Markov game into an altar-augmented setting
• Demonstrates that the presence of explicit classification institutions, such as the altar, improves agents’ ability to achieve cooperation, enforce norms accurately, and adapt effectively to uncertainty.
• Showcases the utility of multi-agent reinforcement learning (MARL) as a tool for investigating complex normative challenges, while emphasizing the critical importance of incorporating institutional structures into environments designed to simulate and study socio economic phenomena.

[1] Punishment allows the evolution of cooperation (or anything else) in sizable groups,, Robert Boyd and Peter J Richerson, Ethology and sociobiology, 1992.