The Meta-Representation Hypothesis

Dear Reviewer twjt,

Thank you for your positive assessment of our work. Below, we will address your concerns.

I think prior work in two related areas should be discussed:

• Invariant representation learning [1, 2].
• Representation learning in RL [3].

Thank you for your valuable suggestion. As the rebuttal process does not permit submitting revised PDF files, we will address these two aspects in the related work section of our future extended version.

The theoretical part is a little hard to follow. In particular, I do not really get the necessity of introducing the first-order approximation $L_ {\pi}$ as in TRPO.

We would be happy to briefly explain for you. We first define the training performance

$\eta(\pi)=\frac{1}{1-\gamma}\mathbb{E}_ {f\sim p_{\mathrm{train}}(\cdot),s\sim d^{\mu_f}(\cdot),a\sim\mu_f(\cdot|s)}[r(s,a)].$

Next, according to the performance difference lemma, we have

$\eta(\tilde{\pi})=\eta(\pi)+\frac{1}{1-\gamma}\mathbb{E}_ {f\sim p_{\mathrm{train}}(\cdot),s\sim d^{\tilde{\mu}_f}(\cdot),a\sim\tilde{\mu}_f(\cdot|s)}[A^{\mu_f}(s,a)].$

Therefore, our objective is to obtain an updated policy $\tilde{\pi}$ based on $\pi$. However, the state distribution $s\sim d^{\tilde{\mu}_f}(\cdot)$ and action distribution $a\sim\tilde{\mu}_f(\cdot|s)$ in their performance difference are all sampled from the new policy $\tilde{\pi}$, where $\tilde{\mu}_f(\cdot|s)=\tilde{\pi}(\cdot|f(s)),\forall f\in\mathcal{F}$, which is clearly infeasible since $\tilde{\pi}$ is unknown at this stage.

Therefore, we must approximate the state distribution (replace $\tilde{\pi}$), which naturally leads to

$L_ {\pi}(\tilde{\pi})=\eta(\pi)+\frac{1}{1-\gamma}\mathbb{E}_ {f\sim p_{\mathrm{train}}(\cdot),s\sim d^{\mu_f}(\cdot),a\sim\tilde{\mu}_f(\cdot|s)}[A^{\mu_f}(s,a)].$

I think the second point is more interesting---although DML itself is not new, to my knowledge, applying similar ideas to RL by simultaneously training multiple agents and letting them mutually regularize each other's policies to improve generalization is novel.

In the original DML paper, different models are trained on the data with the same distribution. Yet in this work, different agents mostly sample different MDPs with different state distributions (this is also reflected in the motivating example in Appendix B). The authors may consider discussing this point in more detail.

Thank you for your insightful comments and suggestions—this is indeed the core motivation and a key distinction from the original DML. We will highlight this distinction in subsequent versions by further expanding Appendix B.

What is the relation between "meta-representations" and invariant representations in the literature?

This is a good question. As you mentioned in the two papers [2, 3], similar concepts have indeed been explored. However, while these approaches typically decouple the generalization problem into robust representation learning (encoder $\phi$) and downstream policy learning, our method is theoretically and empirically end-to-end.

Note that our entire paper never introduces an upstream encoder $\phi$, and we find our approach both more elegant and simpler, as it removes the need to decouple the encoder $\phi$'s learning process from the reinforcement learning process, everything is end-to-end.

Ultimately, while meta-representations and invariant representations are conceptually similar, we argue that the notion of meta-representations is more general. This is because the agent does not need to explicitly learn a robust encoder; instead, the entire end-to-end policy only needs to improve robustness to irrelevant features. Consequently, components beyond the upstream encoder (such as downstream MLPs in the policy network's architecture) may also contribute to this robustness.

Best,

Authors

---

Reference:

[1] M Arjovsky et al. Invariant risk minimization.

[2] A Sonar et al. Invariant policy optimization: Towards stronger generalization in reinforcement learning.

[3] A Zhang et al. Learning invariant representations for reinforcement learning without reconstruction.

Dear Reviewer Rz5E,

Thank you for your positive feedback on our paper. Below, we will address your concerns.

The only concern I have here is that the evaluation is limited to only specific kinds of distributions of rendering functions.

I, however, found that the evaluation in the paper is limited to a narrow set of rendering function changes. I believe this limits the significance of the experiments in the paper. Major changes in the rendering function are not considered. E.g. what if in a game the sprite is changed across episodes? In this sense, I find that the experiments are talking more about robustness to noise, rather than generalization across a wide variety of rendering functions. The paper does allude to generalization of this kind in all sections except the experiment section.

Can this be evaluated under a broader distribution of rendering functions?

Thank you for your insightful comments. However, the observation that our experiments were evaluated under limited rendering functions might be a misunderstanding.

Please note that the Procgen environment samples from a nearly infinite pool of levels when testing generalization performance (each level can be considered as a specific rendering function, since the task semantics remain unchanged). We would like to quote the first paragraph of Section 2 in the original paper of Procgen [1]:

The Procgen Benchmark consists of 16 unique environments designed to measure both sample efficiency and generalization in reinforcement learning. These environments greatly benefit from the use of procedural content generation—the algorithmic creation of a near-infinite supply of highly randomized content. In these environments, employing procedural generation is far more effective than relying on fixed, human-designed content.

You may also refer to our anonymous website at https://dml-rl.github.io/. In the Coinrun environment specifically, the sprite do change across different episodes. Our generalization curves are indeed tested through sampling from a near-infinite set of rendering functions: during training, agents only have access to a limited set of rendering functions, i.e., the first 500 levels, while generalization performance is evaluated across an infinite number of levels. We hope this addresses your core concerns.

Best,

Authors

---

Reference:

[1] K Cobbe et al. Leveraging procedural generation to benchmark reinforcement learning.

Dear Reviewer ByPb,

Thank you for your careful evaluation on our paper. Below, we will address your concerns.

The idea of RL + DML appears to be not very novel. Therefore I believe the key novelty is mainly about how one can get perturbed MDPs. However, I tend to think that the randomized CNN approach is a bit too simplistic.

The concept of DML is not novel and was used in supervised learning. However, although our core algorithm is based on existing methods, the DML used in this paper is with strong motivation and different from the original one. We would like to quote reviewer twjt's comment: "Although DML itself is not new, to my knowledge, applying similar ideas to RL by simultaneously training multiple agents and letting them mutually regularize each other's policies to improve generalization is novel."

Moreover, we conducted an in-depth analysis of why DML is effective and provided both theoretical and empirical evidence demonstrating its generalization benefits, thereby constituting our additional contribution.

Only evaluates PPO. It could help to have a few other RL algorithms.

Thank you for your suggestion. We have added two additional baselines, SPO and PPG. We have the following generalization performance results:

Considering that PPG is an enhanced version of PPO that incorporates additional distillation strategies with relatively complex implementation, we specifically compare the generalization performance between standard PPG and PPG utilizing the frozen encoder obtained from PPO with DML:

More details: https://anonymous.4open.science/r/meta-hypothesis-rebuttal-C70B/README.md

Why are existing representation learning approaches not worthy candidates for comparison as encoders?

Thanks. Indeed, there are numerous works based on data augmentation. However, data augmentation introduces additional prior knowledge, which inevitably induces biases in the training data. These biases may be either beneficial or harmful to generalization, and they require human prior knowledge to guide the selection of appropriate data augmentation techniques for the given scenario.

Therefore, our approach emphasizes spontaneous learning of the underlying semantics by agents through mutual regularization, which is simple to implement, intrinsically unbiased, and generally applicable to a wide range of scenarios. More importantly, while prior works decouple the generalization problem into robust representation learning and downstream policy learning, our method is theoretically and empirically end-to-end.

It would be useful to analyze the informativeness of the meta-representation... Does it simply erase the textural information from the representation while retaining only edge features? ...

Thank you for your constructive insights. This depends on specific scenarios. The total loss consists of both RL loss and KL regularization loss. If texture features are important for the current task, the agent will learn to preserve them, as removing such features would make the RL loss term harder to optimize, potentially resulting in suboptimal policy performance.

More ablation could help. A simple ablation is to mix the MDPs during training without using mutual learning...

Thank you for your suggestion. However, directly mixing the training data from both agents may not be theoretically sound, as PPO's reinforcement learning loss requires data sampled from the old policy for computation, and the data distributions sampled by the two policies could differ. Therefore, we have included an ablation study with a PPO baseline that uses double the batch size and number of interactions. Due to response length limitations, please refer to the table in our response to reviewer bqXb and our link.

Are there ideas about how to go beyond just CNN-weight randomization for sampling the $f$s?

We appreciate the insightful feedback. Indeed, the real-world variations are challenging to capture via CNN randomization alone. We also tested the robustness of DML under different brightness, contrast, and hue conditions, see https://anonymous.4open.science/r/meta-hypothesis-rebuttal-C70B/README.md

For real-world MDP variation, we could consider: (a) Vary factors (e.g., lighting/angle) in real-world datasets using robotic platforms, e.g., habitat. (b) Leverage physics-aware tools, e.g., Omniverse, to simulate plausible variations (e.g., shadows) on real data.

Best,

Authors

Dear Reviewer bqXb,

Thank you for your constructive feedback on our paper! Below, we will address your concerns.

Did you correct for the "actual" number of environment interactions? Does the DML combo with two policies see double the states compared to the PPO baseline?

Thanks. First, the two policies indeed double the actual number of interactions. However, when computing their respective losses, the two agents cannot directly access each other's training data for RL training. Specifically, assume that agent A (denote as $\pi_ A$) collects a batch of training data $\mathcal{D}_ A=\textbraceleft(o_ 1^A,a_ 1^A,r_ 1^A),\dots,(o_ k^A,a_ k^A,r_ k^A)\textbraceright$, agent B (denote as $\pi_ B$) collects a same batch of training data $\mathcal{D}_ B=\textbraceleft(o_ 1^B,a_ 1^B,r_ 1^B),\dots,(o_ k^B,a_ k^B,r_ k^B)\textbraceright$. Then, the total loss for agent A is

$\mathcal{L}_ {A}=\underbrace{\mathcal{L}_ {\mathrm{RL}}(\mathcal{D}_ A)}_ {\mathrm{\text{RL loss}}}+\underbrace{\frac{1}{k}\sum_ {i=1}^{k}D_ {\mathrm{KL}}(\pi_ B(\cdot|o_ i^A)\Vert\pi_ A(\cdot|o_ i^A))}_ {\mathrm{\text{DML loss}}}$

Here, agent A's total loss $\mathcal{L}_ {A}$ only involves agent A's own dataset $\mathcal{D}_ A$, while the DML loss serves as a regularization term. Therefore, from each agent's own perspective, the number of interactions remains consistent with the baseline.

Could you add an ablation for the number of parameters?

Could you add an ablation for the number of steps...?

We have added the ablation experiments with double parameters. To ensure fairness, we also doubled the batch size and total number of interactions for the PPO baseline. The table below shows the generalization performance across four environments:

Limited evaluation.

We provided an additional RL baseline with our method, please see Fig.2 in our link.

For more results: https://anonymous.4open.science/r/meta-hypothesis-rebuttal-C70B/README.md

Section 6 lists the relevant works, although...

Thanks! We will definitely expand the related work in Section 6 in our extended paper.

The theory assumes a bijection between the real state space and the observation space, which excludes many cases where partial observability collapses multiple states under the same observation. Could you comment on how your theory would extend to Partial observability, or how important the assumption of such a bijection is beyond formalising the problem?

Great question! Mathematically, the POMDP problem is consistent with the theoretical framework presented in this paper. Under the POMDP setting, $s$ represents the global information, while $f$ can be regarded as a masking function that obscures the global state $s$ into partial observations $o=f(s)$. Therefore, if we regard the underlying state $s$ in this work as global information and the rendering function $f$ as a masking function, then the generalization problem can be viewed as a POMDP problem.

However, for the POMDP problem, $f$ can only be guaranteed to be surjective but not bijective. For example, for two different global information states $s_ 1$ and $s_ 2$, after applying a certain masking function $f$, $f(s_ 1)=f(s_ 2)$ could happen. In this case, $|\mathcal{O}_ f|<|S|$. Just like two similar maze environments, the global states $s_1\neq s_2$. However, by masking the dissimilar parts of the mazes, the agent's observations become identical. This results in the policy being unable to truly distinguish between these two different global states, i.e., $\pi(\cdot|f(s_1))\equiv\pi(\cdot|f(s_2))$, thus potentially failing to learn the optimal policy.

As a result, the PODMP problem is more challenging than the generalization problem in this paper, as the policy may require additional structures to facilitate learning, such as using a recurrent neural network (RNN) to encode historical information, which inherently introduces non-Markov property.

Best,

Authors