Causality Meets Locality: Provably Generalizable and Scalable Policy Learning for Networked Systems

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of theoretical rigor. We would like to address your concerns/comments as follows.

W1/Q1: "how much value the ACR can bring" "example of a realistic problem where this reduces dimensionality"

In our wireless benchmark, the local state of each user $i$ is represented as $s_i(t) = (b_{i,1}(t), \ldots, b_{i,d_i}(t), z_{i,1}(t), z_{i,2}(t))$. $b_i(t) \in \\{0,1\\}^{d_i}$ encodes queue status by deadline bins (e.g., $b_i(t)=(1,1,0)$ means the first two deadline bins are occupied while the third is empty). $z_{i,1}(t)$ and $z_{i,2}(t)$ are irrelevant components to decision quality:

• $z_{i,1}(t)$: channel quality indicator, which does not influence the packet success probability $q_i$, as $q_i$ depends solely on collisions.
• $z_{i,2}(t)$: grid coordinates, which adds no useful information because the connectivity graph already encodes all relevant interactions.

The ACR procedure automatically prunes such irrelevant or redundant variables, retaining only those that causally influence the local reward or dynamics. This yields a non-trivial reduction in local dimensionality. For example, on a $4\times4$ grid with $d_i=2$ and a neighborhood size $|N_i|=5$:

• Raw (truncation-only): $|N_i|\times(d_i+2)=5\times4=20$ features.
• With ACR: $|N_i|\times d_i=5\times2=10$ features.

This 50% shrinkage directly translates into:

• Smaller critics and lower compute: reduced local state–action support and table size.
• Faster learning: lower variance in actor updates and tighter constants in our bounds.
• Accuracy preserved: approximation error remains controlled (see Prop. 1–2).
• Improved sample efficiency: by effectively increasing the minimum visitation $\sigma$ and reducing the mixing constant $\tau$ (Assumption 2), ACRs tighten the constants in the convergence guarantee (Thm 2).

More generally, ACR is valuable when:

• State representations include auxiliary information, e.g., sensor readings not directly affecting rewards
• Network structure creates redundancies, e.g., traffic flow where only upstream neighbors matter
• Domain factors have sparse influence, e.g., weather affecting only outdoor equipment in power grids

We have included this concrete wireless example, and highlighted which variables are pruned and by how much, along with mask sparsity and critic-size statistics, in the revised paper.

W2/Q3: "crucial assumption required for domain factor estimation" "how to modifify to handle the case with unknown $f_{i,j}$"

We would like to clarify that our approach does not assume explicit knowledge of the closed-form dynamics $f_{i,j}$. We estimate domain factors using observed trajectories and causal masks (see Section 5, Lines 226–233). The structural matrices $c_{\cdot \to \cdot}$ are identifiable under standard causal discovery assumptions (Thm 1). As shown in Appendix F (Prop. 4 and 5), the domain factor $\omega$ are estimated from observed transitions using a maximum likelihood estimation. This likelihood can be evaluated using a learned parametric approximation $\hat{f}$ rather than the true $f$. Thus, our analysis and results remain valid for unknown dynamics, provided that $\hat{f}$ is a consistent estimator of the transition function. Furthermore, Prop. 5 in Appendix F ensures that as long as $\hat{f}$ approximates $f$ within $\epsilon$, the error in domain factor estimation increases by at most $O(\epsilon)$ due to the Lipschitz continuity of the transition likelihood. To avoid confusion, we have clarified these details in the revised version.

W3: only applies to finite state and action spaces

Scope. Thank you for raising this point. Our contribution targets the gap between provably scalable, localized MARL on network graphs (where most analyses are tabular/finite) and cross-domain generalization/adaptation. To the best of our knowledge, we provide the first unified algorithm and theoretical analysis that couples causal locality (ACRs) with domain-factor conditioning and delivers finite-sample convergence and adaptation guarantees for networked systems. We instantiate the theory in finite spaces to keep assumptions transparent and to isolate the impact of ACRs and meta-conditioning without function-approximation confounders.

Extensions. The ACR and GSAC framework extend naturally to continuous spaces by (i) replacing tabular modules with function approximators, and (ii) using entropy-regularized localized actor–critic updates. Because ACRs prune inputs rather than restrict the function class, the variance/conditioning benefits remain, and the exponential-decay/causal-mask assumptions carry over under mild Lipschitz/mixing conditions. In this paper we focus on the finite-space instantiation and position the continuous extension as complementary future work. In the revision we have added a concise discussion of assumptions, connections to continuous MARL, and implementation pointers, and we explicitly cite representative continuous MARL works to position our approach as complementary.

W4: only one benchmark, the latent domain factor is quite simple

The wireless environment is widely used as a stress test for decentralized MARL algorithms [1]. Despite its simple domain factor (packet arrival probability $p_i$), the interaction graph and collision dynamics induce highly nontrivial coordination challenges, especially as grid size grows (Fig 2). That said, we agree that broader empirical validation would strengthen the work.

• Add benchmark: we extend experiments to traffic control network [1,2] on a road-link graph, where state per link is queue counts, action controlling the on-off of turn movement, and reward is negative total queue length. The preliminary results already show consistent advatanges (much faster adaptation compared to $`LFS`$).

• Add baselines: we implement two additional baselines, (i) a multi-task variant of GSAC, (ii) a domain-factor-free baseline. We put the detailed experimental results and discussions to the response to Q4.

W5: typos

Thank you for pointing this out. We have corrected this typo and carefully proofread the paper.

Q2: how the sample complexity of adapting to a new domain depends on the similarity of the new domain to earlier domains seen in training?

Theorem 4 bounds the adaptation gap as $O(1/\sqrt{T_a})$. The key idea of proving Theorem 4 is the inequality $J(\theta(K),\hat{\omega}^{M+1};\omega^{M+1})-J(\theta(K),\omega^{M+1};\omega^{M+1}) \ge -L_{\omega'} || \hat{\omega}^{M+1}-\omega^{M+1} ||.$ Thus, when target domains are close to training domains, the total variation distance $|| \hat{\omega}^{M+1}-\omega^{M+1} ||$ is smaller. This leads to smaller adaptation gap, resulting in faster adaptation. We have added this discussion in the revised version.

Q4: compare against the controller trained with a variety of domains, and then allowed to fine-tune on the new domain data, without explicitly conditioning the policy on the domain factor

Thank you for this excellent suggestion. Following your and ReviewerDb8N' suggestions, we add two new baseliness:

• $`SAC-MTL`$ (multi-task): learns $\pi_{\theta}(a|s,z)$, where $z$ indicates the onehot encoding of the task; optimized jointly over source domains [5].
• $`SAC-FT`$ (fine-tune, suggested by Reviewer y5CC): trains a single policy $\pi_{\theta}(a|s)$ across sources without domain-factor conditioning and fine-tunes $\theta$.
• $`GSAC`$ (ours): learns $\pi_{\theta}(a|s,\omega)$, and adapts by adjusting $\omega$ (estimated from a few trajectories), not $\theta$

Return evaluation on a 4×4 network (source domain $p_i \in \\{0.2, 0.5, 0.8\\}$, $p_{target}=0.65$, higher is better). “Ep” denotes adaptation episodes collected in the target domain.

Initially, performance ranks $`GSAC`$>$`SAC-FT`$>$`SAC-MTL`$>$`LFS`$, gaps that persist through Ep 50, highlighting sizable initial gains over $`LFS`$ (+45% for $`GSAC`$, +40% for $`SAC-FT`$, +11% for $`SAC-MTL`$). $`GSAC`$ achieves the best few-shot performance (Ep 1–50), reflecting rapid adaptation from minimal target data. $`SAC-MTL`$ overtakes after ~130 episodes and remains best thereafter. $`SAC-FT`$ outperforms $`SAC-MTL`$ initially but lags it later, and surpasses $`GSAC`$ after ~250 episodes. $`LFS`$ improves the slowest and only exceeds $`GSAC`$ after ~300 episodes. This matches our design goal: optimize early-episode adaptation by adjusting $\omega$ rather than re-training $\theta$.

References

[1] Guannan Qu, et al. Scalable reinforcement learning for multi-agent networked systems. Operations Research, 2022.

[2] Varaiya, P. Max pressure control of a network of signalized intersections. Transportation Research Part C: Emerging Technologies, 36, 177-195, 2013.

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of theoretical rigor and clear and structured algorithm. We would like to address your concerns/comments as follows.

W1: computational overhead of causal discovery and ACR construction

We acknowledge that Phase 1 (causal discovery) and Phase 2 (ACR construction) introduce upfront costs. However, these phases only incur one-time, local, and amortized cost.

• They are one-time preprocessing steps for each source domain and do not need to be repeated during meta-training or adaptation.
• Both steps are local (per agent and over small neighborhoods), parallel across agents, and the results are re-used for the entire meta-training horizon and for adaptation. We make this explicit in Algorithm 1 (Phases 1–2 precede meta-training and adaptation).

In addition, ACRs reduce overall cost. As shown in Section 3 and Theorem 2–4, ACRs reduce the effective input dimensionality from $O(|s_{N^k_i}|)$ to $O(|s_{N^{\kappa}_i}|)$, resulting in quasi-linear scaling in the number of agents. This leads to smaller mixing times $\tau$ and higher minimum visitation probabilities $\sigma$, improving both convergence speed and sample efficiency (see Section 6.3). Overall, while causal discovery and ACR construction are non-trivial, they are (i) local/parallel, (ii) one-time per source domain, and (iii) reduce the total training/adaptation cost via smaller representations and better constants in the convergence bounds.

W2: seems mainly theoretical and only has one experiment (and no baselines)

Our primary contribution was to establish the theoretical foundations of GSAC, including the structural identifiability, convergence bounds, and adaptation guarantees. However, we admit that broader empirical coverage will strengthen the paper. Here are clarifications:

• What is already included: Appendix H details multiple grid sizes (3×3, 4×4, 5×5), target-domain sweeps for the target packet-arrival rate, and a Learning-From-Scratch ($`LFS`$) baseline trained directly in the target domain. Across settings, GSAC adapts in far fewer steps and outperforms $`LFS`$ (see Fig. 1b, 4–6)
• Add benchmark: we extend experiments to traffic control network [1,2] on a road-link graph, where state per link is queue counts, action controlling the on-off of turn movement, and reward is negative total queue length. The preliminary results already show consistent advatanges (much faster adaptation compared to $`LFS`$).

• Add baselines: we implement two additional baselines, (i) a multi-task variant of GSAC, (ii) a domain-factor-free baseline. We put the experimental results and discussions to the response to Q3.

W3: "Relies on faithfulness, sub-Gaussian noise, bounded degree, full observability, and Lipschitz continuity (Proposition 4)—conditions that may not hold in many real systems"

Our assumptions in Proposition 4 are standard in causal discovery and theoretical MARL.

• Faithfulness and sub-Gaussian noise: these are classical and standard requirements to ensure identifiability [3,4] and finite-sample recovery.
• Bounded degree: bounded neighborhood degree is natural in networked systems where agents have limited communication range. For example, this holds when each intersection connects to only a few roads, or each wireless node has limited interference range.
• Full observability: while we currently assume full local observability, the ACR construction can be extended to partially observable settings using learned belief states or latent encoders. This is part of our future work.
• Lipschitz continuity: this mild smoothness assumption is satisfied by most smooth policies.

Importantly, our algorithm can still be applied when assumptions are violated, only the theoretical guarantees may not hold exactly. We have added a “Limitations and assumptions” paragraph to discuss where these conditions may fail and how GSAC could be extended.

Q1: layers/parameters of the policy

In the wireless benchmark the local action space is discrete, and we do not use a neural network for the policy. We use a softmax (Boltzmann) policy over local actions conditioned on ACR inputs: $\pi(a|s,\omega) = \text{softmax}( \theta[a]^\top (s,\omega)/\tau ),$ where $\theta\in\mathbb{R}^{|A|\times d}$ is the parameter matrix, $(s,\omega)\in\mathbb{R}^{d}$ are the ACR features, and $\tau=0.5$ is the softmax temperature.

Q2: Why not try this on robotic simulation, such as Metaworld?

Our focus is on is large-scale networked systems (traffic, power, wireless), where agents interact over a graph and where $\kappa$-hop truncation and local causal masks are principled. Metaworld [5] tasks lack the explicit networked graph structure that GSAC is designed to exploit. It is primarily single-agent, and running multiple independent MetaWorld tasks in parallel does not induce the structured inter-agent dynamics our theory leverages. Our theorems and ACR constructions are explicitly graph- and neighborhood-based. That said, applying GSAC to robotic settings with explicit interaction graphs (e.g., multi-robot navigation with communication/collision constraints or multi-arm coordination with shared resources) is a natural and valuable next step. We have added discussion of this direction in the revised paper.

Q3: Why meta learning as opposed to multitask learning? Are there ablations regarding this?

Clarification of meta-RL vs. multi-task RL paradym: the goal of multi-task RL [5] is to learn a single, task conditioned policy that maximize the average expected return across all source domains from a domain distribution. However, it offers no guarantee of rapid few-shot adaptation from new, unseen domain. Meta-RL, in contrast, explicitly learns to adapt: it leverage the set of source domains to learn a policy that, at test time, can quickly adapt to a new domain with few trajectories.

Why meta-learning: our goal is is fast adaptation in a new domain from few trajectories. We train a shared, domain-factor-conditioned policy across sources and, at test time, estimate domain factors $\hat\omega$ from a few trajectories to adapt immediately. Our theory shows:

• Meta-gradient error terms shrink with the number of source domains $M$ and the samples used to estimate domain factors $T_e$.
• The adaptation gap decays as $O(1/\sqrt{T_a})$ with the number of adaptation trajectories.

New baselines:

• $`SAC-MTL`$ (multi-task): learns $\pi_{\theta}(a|s,z)$, where $z$ indicates the onehot encoding of the task; optimized jointly over source domains [5].
• $`SAC-FT`$ (fine-tune, suggested by Reviewer y5CC): trains a single policy $\pi_{\theta}(a|s)$ across sources without domain-factor conditioning and fine-tunes $\theta$.
• $`GSAC`$ (ours): learns $\pi_{\theta}(a|s,\omega)$, and adapts by adjusting $\omega$ (estimated from a few trajectories), not $\theta$

Return evaluation on a 4×4 network (source domain $p_i \in \\{0.2, 0.5, 0.8\\}$, $p_{target}=0.65$, higher is better). “Ep” denotes adaptation episodes collected in the target domain.

Initially, performance ranks $`GSAC`$>$`SAC-FT`$>$`SAC-MTL`$>$`LFS`$, gaps that persist through Ep 50, highlighting sizable initial gains over $`LFS`$ (+45% for $`GSAC`$, +40% for $`SAC-FT`$, +11% for $`SAC-MTL`$). $`GSAC`$ achieves the best few-shot performance (Ep 1–50), reflecting rapid adaptation from minimal target data. $`SAC-MTL`$ overtakes after ~130 episodes and remains best thereafter. $`SAC-FT`$ outperforms $`SAC-MTL`$ initially but lags it later, and surpasses $`GSAC`$ after ~250 episodes. $`LFS`$ improves the slowest and only exceeds $`GSAC`$ after ~300 episodes. This matches our design goal: optimize early-episode adaptation by adjusting $\omega$ rather than re-training $\theta$.

References

[1] Varaiya, P. Max pressure control of a network of signalized intersections. Transportation Research Part C: Emerging Technologies, 36, 177-195, 2013.

[2] Guannan Qu, et al. Scalable reinforcement learning for multi-agent networked systems. Operations Research, 2022.

[3] J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, 2000.

[4] Huang, Biwei, et al. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729 (2021).

[5] Tianhe Yu, et al. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. Proceedings of the Conference on Robot Learning, PMLR 100:1094-1100, 2020.

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of soundness, originality and significance of our work. We would like to address your concerns/comments as follows.

W1: "add some sort of roadmap in the beginning of the methods section, showing how all these pieces, both theory and the algorithm, fit together into a single story" "In Section 4.1, the authors state the parts of the Algorithm, but it would be also good if they linked these statements with the theorems"

We appreciate the suggestion of adding a roadmap theorem-algorithm linkage in Section 4. In the revised version,

• we introduce a high-level roadmap at the start of Section 4, outlining how causal discovery, ACR construction, and meta actor-critic optimization fit together into a unified story (Phases 1–4, Algorithm 1).
• we explicitly link the algorithmic components with the corresponding theoretical results (Theorems 1–4 and Propositions 1–5).
• we also add a brief summary figure (Figure 1) to visually illustrate the GSAC pipeline, complementing the roadmap.

We add Roadmap and Theoretical Connections at the start of Section 4 (right before Section 4.1, Algorithm overview). It combines the roadmap with explicit theorem-algorithm linkages:

Our GSAC framework (Algorithm 1) integrates causal discovery, representation learning, and meta actor-critic optimization into a unified pipeline, with each phase supported by theoretical results. Phase 1 (causal discovery and domain factor estimation) is underpinned by Theorem 1 and Propositions 4–5, which establish structural identifiability and sample complexity guarantees for recovering causal masks and latent domain factors. Phase 2 (construction of ACRs) leverages the causal structure to build compact representations of value functions and policies, with bounded approximation errors rigorously characterized by Propositions 1–3. Phase 3 (meta actor-critic learning) performs scalable policy optimization across multiple source domains, with convergence of the critic and actor updates guaranteed by Theorem 2 (critic error bound) and Theorem 3 (policy gradient convergence). Finally, Phase 4 (fast adaptation to new domains) exploits the learned meta-policy and compact domain factors to achieve rapid adaptation, where the adaptation performance gap is formally controlled by Theorem 4. Together, these results demonstrate that each algorithmic component is theoretically justified and collectively leads to provable scalability and generalization in networked MARL.

We also add a one-paragraph explanation referring to Figure 1:

Figure 1 complements Algorithm 1 by showing how causal recovery and ACR construction (Phases 1–2) compress the state-action space, enabling efficient meta-policy training (Phase 3), and how this meta-policy rapidly adapts to new domains (Phase 4), as guaranteed by Theorems 2–4.

Q1: citation of the empirical task

Thank you for pointing it out. We have updated our references to cite the original work [1] on the multi-channel CSMA network model in Section 7 Numerical experiments, alongside the benchmark paper [2]. We also added references to cite this paper in Introduction on wireless communication systems.

Limitations: "in which ways could the assumptions 1-6 not to be met in practice, and what would be the implications for the proposed method"

You raised an important point regarding Assumptions 1–6 (line 256). A1-A4 are standard assumptions for proving convergence of networked MARL algorithms, without consideration of domain generalization/adaptation [2-4]. For example, [4] provides the first provably efficient MARL framework for networked systems under the discounted reward setting, and including the first convergence analysis of networked MARL algorithm under A1-A4.

Our work further fills the gap between scalability and generalizability across domains in networked systems. To this aim, we introduce additional asumptions A5-A6 regarding the latent domain factor. A5 is similar with A3, which imposes the smoothness w.r.t. the domain factor $\omega$, while A3 imposes the smoothness w.r.t. the actor parameter $\theta$. A6 is a regularity assmption to ensure the compactness of domain factor space.

While A1-A6 are standard for establishing theoretical guarantees, we discuss whether they hold exactly in practice as follows:

• Finite MDP (A1): in some systems, the state or action spaces might not finite or continuous. For such cases, our results can still extend by applying state aggregation or clipping methods to ensure boundedness while incurring small approximation error.
• Mixing properties (A2): real-world networked systems may exhibit slow mixing or non-Lipschitz dynamics. Our theoretical results would degrade gracefully, leading to larger constants $\tau$ in the sample complexity bounds.
• Smoothness (A3): the assumption requires the policy gradients $\nabla_{\theta_i} \log \pi_i$ to be Lipschitz and uniformly bounded. This holds for all commonly used policy parameterizations (e.g., softmax policies) with bounded inputs and finite-dimensional parameter vectors. It ensures stable policy updates and is common in the convergence analysis of policy gradient and actor-critic algorithms. $`GSAC`$ uses softmax parameterizations for policies, where $\log \pi_{\theta_i}$ is smooth and has bounded gradients when action probabilities are not close to zero (a condition satisfied by entropy regularization or initialization).
• Compact parameter space (A4): policy parameters $\theta_i$ are always constrained within a finite range due to neural network weight regularization, weight clipping, or bounded initialization. This is standard in actor-critic methods, where optimization is performed with bounded learning rates and regularizers that implicitly keep the parameters within a compact set.
• Smoothness w.r.t. domain factor (A5): this is reasonable because domain factors, e.g., channel gains, traffic loads, represent physical parameters that vary smoothly. The local dynamics $P_i$ are typically analytic or polynomial in $\omega_i$, so small perturbations in $\omega_i$ induce proportionally small changes in state transitions and rewards. Consequently, the Q-function $Q^\pi_i(s,a,\omega)$ and policy gradient $\nabla_{\theta_i} J(\theta,\omega)$ inherit this smoothness. Moreover, standard policy parameterizations and neural network approximators are Lipschitz when combined with bounded weights or gradient clipping. Thus, A5 aligns well with both the structure of real-world systems and common RL practices. Moreover, even if the true dynamics were not strictly Lipschitz, domain factor estimation (Proposition 5) uses a small-perturbation assumption around $\omega$ for adaptation. In practice, state or reward clipping and function approximation, which are inherently Lipschitz, effectively enforce this assumption.
• Compact domain factor space (A6): the latent domain factors $\omega_i$ represent bounded variations in environment dynamics (e.g., channel gains, traffic intensities, or reward scaling factors), which are inherently limited by the physical system. In real-world networked systems, these factors are naturally confined to finite ranges determined by system design (e.g., transmission powers, queue lengths). Hence, assuming $\Omega_i$ is compact with bounded diameter $D_\Omega$ is both reasonable and practical.

In the revised version, we have added discusssions about these potential limitations and their implications in Section 8 and Appendix G, making clear how they might affect the performance of $`GSAC`$ in less structured or highly stochastic environments.

Additional minor comments

We have corrected the typo at Line 220 (“weights them” → “weighs them”).

References

[1] Alessandro Zocca. Temporal starvation in multi-channel csma networks: an analytical framework. Queueing Systems, 91(3-4):241–263, 2019.

[2] Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learning for networked systems with average reward. Advances in Neural Information Processing systems, 33:2074–2086, 2020.

[3] Yiheng Lin, Guannan Qu, Longbo Huang, and Adam Wierman. Multi-agent reinforcement learning in stochastic networked systems. Advances in neural information processing systems,34:7825–7837, 2021.

[4] Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning for multi-agent networked systems. Operations Research, 2022.