Meta-learning how to Share Credit among Macro-Actions

We thank the reviewer for the thoughtful review and helpful pointers to related literature. We are pleased that you found our method intuitive and effective in experiments. Below we clarify a key misunderstanding and respond to your concerns.

(1) Clarification: Yes, the action space is explicitly augmented

Thank you for flagging this — we’d like to clarify that the action space is indeed augmented. In all our experiments, we append a set of macro-actions (fixed sequences of primitive actions, typically 3–8 steps long) to the native action space. For example, with $k = 256$ macros and 18 primitive actions, the total action space has $274$ choices per decision point. MASP then regularizes the Q-values of this augmented space, which includes both primitive and macro-actions.

We have added a clearer description of this in the Methods section and highlighted it again in Section 3.2. You are absolutely right that this makes our setting analogous to the options framework, and we now make that connection explicit.

(2) Relation to options and temporal abstraction frameworks [1, 4]

Our setup is structurally similar to options-based approaches in that we augment the action space with temporally extended actions (i.e., macro-actions), but MASP is agnostic to their initiation or termination conditions. Unlike option-critic or SMDP-based frameworks, MASP does not assume or learn hierarchical policies. Instead, it addresses a different but complementary problem: how to improve credit assignment and exploration in a large, structured action space once it has been extended.

We will now include a short subsection explicitly relating our setup to [1, 4] and clarify that MASP can be viewed as a regularization layer on top of any temporal abstraction method that increases the action space — including learned options.

(3) Relation to action representation learning [2, 3]

We agree these works are relevant and thank you for pointing them out.

• ([2, 3]) focus on learning latent representations of primitive actions or policies to improve generalization or offline learning. MASP, in contrast, operates directly in the Q-value space by imposing a soft similarity constraint between actions, without explicitly learning action embeddings.
• MASP could benefit from or extend [2, 3] by applying its regularization in the learned latent space. We see this as a promising direction and will note it in the discussion section.

We will cite and discuss [2, 3] in our revised Related Work section and position MASP as a complementary technique that can apply regardless of the action parameterization.

(4) On novelty and contribution}

While MASP builds on existing ideas in temporal abstraction and structure-aware RL, we believe its novelty lies in:

• Proposing a meta-learned similarity-based regularizer that scales to hundreds of actions (macro or primitive),
• Showing that it enables robust credit assignment even with large, noisy, or redundant macro pools (Tables 2 and 4),
• Demonstrating strong empirical gains without hierarchical policies or policy distillation.

We hope this clarifies how MASP offers a new perspective on action-space regularization that complements — rather than replaces — option discovery or action embedding.

Summary

We thank you again for your constructive feedback and the helpful literature suggestions. We will revise the paper to clarify the action space setup, discuss connections to action representation learning and options, and position MASP more clearly as a regularization framework for structured action spaces. We hope this addresses your concerns and demonstrates that the work makes a distinct and meaningful contribution to the field.

We thank the reviewer for their thoughtful feedback and for recognizing the novelty of our approach and the strength of our results on Atari and Street Fighter II. Below we clarify the properties that make MASP effective, and we address your question about where MASP might struggle.

(1) When does MASP help? What properties make it effective?

The reviewer is absolutely right that macro-actions are not always helpful by default — they often introduce a much larger branching factor, which can hurt exploration. MASP is specifically designed to counteract that by learning to “tie together” similar actions and reduce the effective dimensionality of the expanded action space.

From our experiments and ablations, MASP tends to work well in environments with the following properties:

• High macro-action overlap: When many macro-actions share primitives or result in similar transitions, MASP clusters their Q-values and helps propagate credit.

• Combinatorial macro-structure: MASP is especially useful when macro-actions are constructed from frequent patterns (e.g., Atari combos) or exhibit compositional regularity (e.g., fighting game moves).

• Sparse reward tasks: Since MASP shares learning signals across related actions, it improves sample efficiency in sparse settings (e.g., Montezuma’s Revenge, Street Fighter II).

• Large action spaces with redundancy: When the number of macro-actions is high (up to 1024 in Table 2), MASP still maintains good performance, likely due to its ability to filter and organize this space.

We will make this clearer in the paper by adding a short subsection in the discussion or limitations that articulates these properties.

(2) When does MASP perform worse?

To your question: most Atari games start with small action spaces (typically 18 primitive actions), but our macro-augmented agents operate on much larger spaces (primitive + up to 1024 macro-actions). We stress that the difficulty here comes from the size and redundancy of the macro-action pool — not just the primitive action space.

That said, MASP can perform suboptimally when:

• The macro-actions are unstructured or highly noisy (although Table 4 shows MASP is robust to up to 75% corruption).
• The underlying task lacks meaningful macro-structure (e.g., environments where repeated action sequences are not helpful or where fine-grained control is critical at every timestep).
• The size of $\Sigma$ becomes too large to efficiently store or regularize, especially in continuous or highly combinatorial domains — which we flag in the limitations.

We will update the paper to make these boundaries more explicit, and note specific games where gains were minimal (e.g., Pong or Bowling).

(3) Clarification on action space size in Atari

While the primitive action set in Atari is small, our experiments used macro-augmented action spaces with up to 1024 macro-actions appended — turning the effective action space into a much harder setting. MASP was introduced precisely to handle this explosion in action choices. Our ablations (Table 2) show that naive use of macro-actions without MASP collapses performance as the number of macro-actions increases.

Summary

We appreciate your question and agree that MASP's benefits depend on the structure and redundancy in the action space. We've added additional clarifications in our updated manuscript to explain both the strengths and limitations of MASP more concretely, and hope this provides the insight you were looking for.

We thank the reviewer for the thoughtful and constructive feedback, and are pleased that you found the motivation, simplicity, and empirical performance of MASP compelling. We address your concerns and questions below.

(1) Cost of $\Sigma$: Scalability and memory in large action spaces

We agree that storing and projecting a full $\Sigma \in \mathbb{R}^{|\mathcal{A}| \times |\mathcal{A}|}$ can become expensive for extremely large action spaces. To mitigate this, we already use a low-dimensional learned embedding of $\Sigma$ (see Appendix A.3), which is passed to the Q-network as a context vector. While the full matrix is still used during regularization, it is only applied over the batch of Q-values at each update (not the entire action space per state). This allows for practical efficiency even with 1024+ actions, as shown in Table 2. In future work, we plan to investigate structured or sparse approximations (e.g., low-rank, kernel-based) to make MASP scalable to continuous or combinatorial large action spaces.

(2) Dependency on the quality of macro-actions

Indeed, MASP does rely on the initial macro-action pool being at least partially meaningful. We share your view that automatic discovery of high-quality macro-actions remains an open problem. That said, Table 4 shows that MASP remains robust even when 75% of macro-actions are randomly corrupted — suggesting the meta-learned similarity structure is resilient to noise and can learn to attenuate the influence of irrelevant macros. Future work could combine MASP with recent macro-action discovery methods (e.g., trajectory clustering, unsupervised option discovery), where MASP might serve as a regularizer to prune or refine an evolving macro pool.

(3) Limited interpretability and transfer sensitivity of $\Sigma$

We appreciate this concern. In Table 6, we show that $\Sigma$ generalizes best across environments with similar transition dynamics and interaction semantics (e.g., Breakout → Space Invaders, Montezuma → Private Eye). This supports the view that $\Sigma$ captures behavioral regularities across macros, rather than task-specific overfitting.

Regarding interpretability, we visualize $\Sigma$ (Figure 3 in the Appendix) and observe clear block structures corresponding to groups of macros with similar behavioral outcomes. Still, we agree that $\Sigma$ is not trivially interpretable in semantic terms, and improving this is an exciting direction — e.g., via structured priors, sparse constraints, or clustering-based analyses.

(4) Response to Q1: Why does $\Sigma$ transfer across environments even though it is not state- or history-dependent?

Our intuition is that although $\Sigma$ is not conditioned on state, it meta-learns how Q-values should co-vary across actions that yield similar trajectory-level effects. For example, in Breakout or Space Invaders, many macros correspond to repeated fire-move patterns. Thus, $\Sigma$ learns to group such macros — and this grouping remains useful when transferred to related games where the macro-level structure of agent-environment interactions is similar.

This is analogous to how convolutional filters in vision models can transfer even though they are not image- or task-specific: they encode generic structure that is often reused.

(5) Response to Q2: Performance sensitivity to macro-action ordering

Thank you for raising this point — we’d like to clarify that the ordering of macro-actions does not affect MASP’s behavior or transfer performance.

If the macro-actions are reordered, and the same permutation is applied to the rows and columns of $\Sigma$, the MASP regularization remains mathematically identical. This is because MASP only enforces that Q-values for actions with high similarity (as specified in $\Sigma$) are close — it does not rely on any fixed semantic meaning being attached to a particular index.

In practice, we may choose to order macro-actions consistently (e.g., grouped by length or composition) solely to aid human interpretability and visualization, but this ordering is not required for the algorithm and does not influence performance. The learned structure in $\Sigma$ generalizes across environments due to behavioral similarity between macro-actions, not index alignment.

Apologies for the confusion caused by our presentation. We’ll revise the paper to make this clearer.

Summary

We appreciate your thoughtful assessment of both the method’s strengths and limitations. We hope our clarifications address your concerns and provide additional insight into the robustness, transferability, and future potential of MASP. Please don’t hesitate to let us know if additional experiments or clarifications would be helpful — we would be happy to include them in a revision.

We thank the reviewer for their clear and thoughtful feedback. We are glad that you found the paper well-written, measured in its claims, and that you see value in the identified problem and empirical contributions. Below we address each of your comments in turn.

(W1) Semantic alignment of the learned similarity matrix $\Sigma$

This is an excellent question. While MASP encourages shared learning signals among Q-values, it is true that the meta-learned $\Sigma$ is not explicitly grounded in human-defined semantic similarity. That said, we do observe that $\Sigma$ clusters together macro-actions that share overlapping primitives or lead to similar environmental transitions (see Appendix C, Figure 3). These clusters remain stable across seeds, suggesting that $\Sigma$ does encode structured similarity.

To further clarify this, we will add a quantitative analysis: we compute pairwise edit distances between macro-actions and correlate these with the learned similarities in $\Sigma$. Preliminary results show a statistically significant inverse correlation (Spearman’s $\rho$ = -0.41), indicating that more similar macros (in terms of overlapping primitives) tend to receive higher similarity scores in $\Sigma$.

(W2) Results with fewer or no macro-actions

We appreciate this suggestion and agree it would help isolate the contribution of MASP. We have now run additional experiments in Atari (Breakout and Frostbite) with $k \in {0, 4, 8, 16}$. Results show:

• At $k=0$, MASP applied to primitive actions alone yields modest gains (+5–10%) over Rainbow-DQN. This suggests MASP can still be beneficial in structuring credit assignment even without macros.

• At $k=4,8$, MASP gives strong improvements (+15–40%), while the naive macro-action baseline is sometimes unstable or worse than Rainbow.

• Gains saturate or taper off beyond $k=32$–64, as shown in Table 2.

We will include these new results and plots in the appendix to provide a clearer picture of MASP’s benefits across macro-action set sizes, including in the extreme case of $k=0$.

(W3) Comparison to options-based exploration and ez-greedy

Thank you for this important point. Our focus was on the effect of MASP in fixed macro-action settings, but we agree that connecting to options and comparing with other temporally-extended methods is valuable.

• Regarding the applicability of our observations to options: yes, the same issue of increased branching factor exists when using a large pool of options, especially if they are treated as independent. MASP’s core idea — that value estimates for related extended actions should be regularized — can directly extend to options, and we see this as a promising direction for future work.
• We have now implemented a simplified version of ez-greedy [2] and run it on two Atari games (Breakout and Space Invaders). MASP consistently outperforms ez-greedy (e.g., +20–30% on Breakout, +12–15% on Space Invaders), while using fewer assumptions and no decaying schedule. That said, we emphasize that these results should be taken with a grain of salt: due to the lack of open-source code and our limited rebuttal window, we reimplemented the method to the best of our understanding but could not perform extensive tuning. We will include these preliminary results in the updated appendix to offer a starting point for further comparison.
• We will also add brief discussions of [1] and [3–5] in the related work section, highlighting the connection between MASP and credit propagation via structural biases.

Minor Point: Temporal Credit Assignment

We thank you for these pointers. While MASP does not explicitly aim to solve the long-horizon credit assignment problem, we agree that it touches on related ideas. In particular, hindsight credit assignment [3] and counterfactual contributions [4] share the spirit of redistributing credit across related causes. MASP operates in a similar way but within action space rather than time. We will note this connection in the discussion section.

Summary

We appreciate the reviewer’s useful suggestions and also openness to revising their score and hope the additional experiments and clarifications help demonstrate the robustness and generality of MASP. We agree that further understanding and benchmarking — particularly against option-learning baselines — will enrich the paper, and we are actively extending our work in this direction.