Reward as Observation: Learning Reward-based Policies for Rapid Adaptation

• Limited Real-World Applicability: Training without direct observations restricts the method’s applicability, as real-world scenarios often involve more complex changes beyond observation shifts. This limitation raises concerns about the generalizability of reward-conditioned policies in diverse settings. What are the possible scenarios in the real world where only observation changes within the MDP?
• Scalability Issues: While the reward-based learning approach shows promise in simpler environments like Cartpole, its effectiveness in more complex tasks such as locomotion, manipulation, or humanoid control remains uncertain. Can reward-based policy handle complex locomotion or manipulation tasks? Since this work focuses on understanding transfer benefits, there is concern that it might struggle to learn necessary behaviors before transfer.

W1. Numerous Assumptions. The method makes many assumptions for successful implementation, limiting its adaptability and real-world applicability.

1. Dense and Accurate Rewards. The authors state that rewards must be sufficiently dense and give a good value throughout the state space, yet there is no analysis of how sparse they need to be. This lack of clarity makes it uncertain how sparse or dense rewards can be before training fails. In practice, it likely means that transitions without an informative reward signal contribute minimally to training, making the approach highly sample-inefficient. In realistic settings, feedback is often imperfect or delayed. For example, in robotics, sensors might provide inaccurate readings due to interference or hardware limitations, leading to noisy or inconsistent rewards. Similarly, in environments where rewards are human-generated (like feedback in recommendation systems), subjective or inconsistent responses can introduce noise.
2. Domain Compatibility. The source and target environments should share the same transition dynamics, action and observation spaces, and reward structure. This overlooks many practical cases, where slight discrepancies in dynamics or observation structures are the norm. The authors don’t discuss where these conditions might realistically apply, leaving practical feasibility unexplored.
3. Expert Guidance. The ablation study shows that expert guidance significantly boosts performance, particularly on complex tasks. However, this dependence undermines the claimed benefits of a “reward-only” approach, as it reintroduces standard RL processes the authors aim to bypass. Moreover, training an observation-based expert policy adds a computational overhead, and since training is done online, inference also needs to be run on the expert model.

W2. Loss of Spatial Context. By relying solely on rewards and actions, the method lacks spatial awareness, which is critical in tasks requiring an understanding of position or orientation. For example, in navigation tasks like maze-solving, an agent without spatial context will struggle to differentiate between distinct but similarly rewarding areas, such as identical-looking corridors or dead ends. In manipulation tasks, the lack of positional feedback leads to incorrect actions, like reaching for objects without adjusting for their relative location. Without such spatial cues, the method is limited to simple tasks where positioning doesn’t play a role. The agent’s ability to keep the car centered on the track in the Car Racing environment hinges on the overly-engineered reward function that promotes this behavior. Relying on such finely-tuned rewards is impractical in real-world applications, where crafting reward functions to this level of precision is rarely feasible.

W3. Lack of methodological novelty. The approach lacks innovation, relying on a simple combination of an LSTM and behavior cloning to train PPO. This simplicity offers little advancement over existing techniques and contributes minimally to the field, as both components are well-established in RL. Moreover, the explanation of the method is scarce. Neither the text nor the caption of Figure 3 adequately explains the diagram, making it difficult to interpret. For example, it’s not immediately evident that the ‘options’ in the testing phase of Figure 3 depict the modes where the environment provides the reward and where the reward is learned.

W4. Weak Experimental Evaluation.

1. The experiments are limited to simple environments (Pointmass, Cartpole, and Car Racing), with the authors suggesting that more complex environments with higher state dimensions or sparse rewards are "impossible" for this method. This raises questions about practical utility—if the method can't handle harder, more realistic tasks, its applicability remains unclear.
2. The authors only compare their reward-based policy to a regular observation-based policy, which is supposed to serve as an upper bound. However, they include no comparisons with other established techniques whatsoever, offering no context on how their approach measures up to alternative methods in this setting.
3. While evaluation is done over 50 trials, the number of seeds used for training isn’t specified. Without this, it’s hard to assess the robustness of the method.
4. The authors state that it is essential that some recurrent network is used, and that that a single reward/action pair is not sufficient. However, they provide no analysis or ablation study to clarify the LSTM’s specific contribution. Alternative approaches for maintaining temporal history, like stacking previous rewards and actions, are not explored, leaving it unclear whether the LSTM is genuinely necessary or if simpler methods could achieve comparable results.

W5. Transfer Limitations. The authors attempt 2D-to-3D transfer by using the Car Racing and AirSim environments, as shown in Figure 1, considering a scenario where changes only occur in pixel-based observations. This leads them to circumvent the realistic challenges of transfer by manually reimplementing the kinematics to match across environments. This undermines the notion of true zero-shot transfer, as manual alignment is rarely viable in real-world applications and reveals the method’s limited applicability. Furthermore, no quantitative results are provided for the 3D transfer; it’s merely claimed that the policy shows “reasonable driving performance,” leaving the success of this transfer largely unsubstantiated. The authors’ statement that “our focus is not on dealing with the dynamics shift” dismisses the real complexities of 2D-to-3D transfer, where adapting to dynamic differences is unavoidable. This approach fails to demonstrate genuine transfer.

I think the major problem with the paper is that the studied setting is very unrealistic. The paper provides some discussions, which also acknowledge how difficult it is for their assumptions to be satisfied. At the end, it is still difficult to think of any application where the proposed approach is going to be useful.

Specifically, I am not worried about the assumption that the reward is observable. But what are some real-world problems / environments where an RL agent can perform reasonably well just by seeing the previous actions and rewards? As the paper acknowledges, this is exactly like trying to play a game by looking only at the score and no other parts of the screen. Other than environments where it is possible to memorize the solution, it seems to me that these conditions are extremely hard to satisfy. And this is not even the only assumption. In addition to these, there should also be some transfer concern where the robot's transition and reward functions stay the same but the observations change. I don't see any useful application.

A somewhat promising part of the paper is the section where it tries to estimate rewards from observations. However, that section requires access to a state-based policy that solves the task. Again, this is very unrealistic. If the problem is a POMDP, how would one have access to a state-based policy? The states are not even known to the agent.

Finally, the main promise of the paper is not very interesting. When the learned policy does not depend on the observations, of course the observation function can be changed in any arbitrary way. Under that setting, that function is completely irrelevant. So the experiments are completely unsurprising and obvious.

Below are my other comments that are more minor:

• et al. is plural, so those citations should be thought as "they" instead of "it". There are some grammatical errors about this.
• The POMDP definition is missing the function that maps the states to observations.
• The so called reward-based policy is defined as $R\times A \to A$ but my understanding is that a full history of rewards and actions are inputted, not just the most recent ones. In that case, this definition of the policy function is incorrect.
• Incorrect capitalization in line 248.
• Again, the second hypothesis/question (line 253) is validated by the definition of reward-based policy. Why is it even a hypothesis?

1. Lack of a justifiable motivation: It's unclear why one would use only rewards as the observation space in practice. For any complex dynamical system, having a policy dependent on an observation is required. Theoretically, this would only work in stationary bandit-like settings, where a single action optimal action exists from the ones available and a history of past actions and rewards would suffice to take the right action. However, in all other “RL” settings where an action taken changes the state of the world, i.e., environments with a transition function, this method would break. Also, it’s hard to motivate this method from a practical perspective - in most real applications, one would utilize all possible information available to learn a behavior. In short, the authors claim they study a much more complex problem - that of reinforcement learning, while applying a primitive set of assumptions - those of a multiarm bandit. I don’t believe this method would work under the advertised conditions. Additionally, I question the claims of generalization in the paper - but of course, the method would generalize across visual observation perturbations. It is not conditioned on these perturbed inputs.
2. Unclear why this works: I suspect the method is able to solve tasks due to the demonstrations available to it - not due to the online PPO. This is clear in the ablations also - for tasks that actually have a transition function like Car Racing, the demonstrations yield appreciable improvements over not using them.
3. Lack of Novelty: It’s unclear to me whether this paper adds to existing knowledge in any way. Multiarmed Bandits are well studied and memory based agents are well studied also. The paper currently does not present any new theoretical insights, nor does it show massively scaled experimentation.