Multi Task Inverse Reinforcement Learning for Common Sense Reward

• Their main method assumes task-specific rewards are given. This is an unrealistic assumption and changes their problem setting from IRL to demonstration-guided RL. In practice their method looks more like learning an auxiliary reward from multi-task demonstrations to supplement the task reward given by the environment.
• The unknown task reward extension of their method naively learns a task specific and common sense reward jointly without any effort at disentangling the two. This will not automatically separate the reward functions and it’s unclear why this method would be better than learning a combined reward function.

There are also some serious issues with the experimental setup, which makes the paper’s main claims unconvincing to me.

• The authors only experiment on selected Meta-World tasks, which are all relatively similar tabletop manipulation tasks. I am not convinced these results are generalizable to other benchmarks and types of tasks.
• Comparison methods. While their method has access to both task specific rewards and expert demonstrations, the baselines methods seem to only have access to one or the other (reward for SAC, demos for IRL methods), so this is an unfair comparison. Furthermore, they are missing other methods that decompose reward functions including Reward Network Distillation which they cite in related work.
• The definition of the target velocity and action norm common sense rewards seem contrived since they are not critical or even necessarily helpful to learning the tasks (since almost all experiments achieve 100% task success). The common sense reward effectively learns a style preference from the demonstrations that’s not present in the task reward function. The setup in section 5.4 where the algorithm extracts both task specific and common sense reward from demonstrations makes much more sense.
• Section 5.4 is missing any baseline methods, especially doing AIRL on each task individually, so I am not convinced that MT-CSIRL+LT is effectively decomposing the reward functions and transferring them.

1. Motivation : The paper's motivation is based on unsubstantiated claims regarding capabilities and limitations of IRL. In line 76-82 the paper claims that "IRL fails to learn correct reward functions due to its connection to GANs". IRL is not equivalent to GAIL; rather GAIL is a way to solve distribution matching and obtained from a primal-dual perspective. While, methods like GAIL are not aiming to learn reward functions there exists method like f-IRL[2] and MaxEntIRL[3] that explicitly try to learn reward functions. f-IRL has looked at learning "stationary reward functions" - reward functions on which the policy can be retrained.
2. Literature review: I found the literature reviews to be somewhat shallow. For example, IRL was attributed to Arora and Doshi (2021); IRL existed long before and its origins can be traced back to [1] and possibly earlier. Authors are encouraged to use correct attributions and perform an in-depth literature review.
3. Unprincipled method: The reward function for any task is assumed to decompose as common sense reward+task-specific reward. Under such a decomposition there is no guarantee that optimizing task-specific reward results in same optimal policy as the combination with common sense reward. Abeel and Ng[4] have proposed a class of reward functions that result in same optimal policy. 4: Assuming known structure of common sense rewards: In experiments it was presumed that common sense reward had a fix structure. I think that is a major limitation as the structure cannot be determined without domain knowledge.
4. Section 4.3 is titled curriculum learning, but I do not see the connection to curriculum learning from the method proposed.
5. Experiments: Standard IRL methods are missing from comparisons like MaxentIRL and f-IRL.

[1]: Abbeel, Pieter, and Andrew Y. Ng. "Apprenticeship learning via inverse reinforcement learning." Proceedings of the twenty-first international conference on Machine learning. 2004. [2]: Ni, Tianwei, et al. "f-irl: Inverse reinforcement learning via state marginal matching." Conference on Robot Learning. PMLR, 2021. [3]: Ziebart, Brian D., et al. "Maximum entropy inverse reinforcement learning." Aaai. Vol. 8. 2008. [4]: Ng, Andrew Y., Daishi Harada, and Stuart Russell. "Policy invariance under reward transformations: Theory and application to reward shaping." Icml. Vol. 99. 1999.

• [Reward structure]: The authors assume an additive reward structure, which may be overly simplistic. It would be beneficial to consider more complex functional forms that are representative of real-world scenarios, such as general non-linear functions. Additionally, incorporating noise factors in both common-sense and task-specific rewards—like additive noise, linear non-Gaussian models, or post-nonlinear models with noise—could enhance the framework. A discussion and analysis of these more general function forms, both theoretically (ensuring that disentanglement factors remain identifiable) and empirically (demonstrating that they can be learned within the framework), would be valuable.

• [Common-sense reward concepts]: The definition of common-sense rewards in this work is quite basic. Can the framework be scaled to learn more generalizable common-sense rewards, such as a foundational understanding of physical world models or other relevant norms applicable to more complex tasks, like those involving embodied agents? Incorporating language might also be a relevant avenue to explore [1]. This point is more of an open question, but any discussion around it would be appreciated.

• [Relation to unsupervised RL] In unsupervised RL, especially mutual-information-based skill discovery, methods can also learn a set of common skills from tasks. It might be beneficial to discuss that line of research and the major benefit of using this framework to learn disentangled common-sense rewards.

• [Evaluation]: Is there any theoretical or formal analysis addressing why common-sense reward learning performs poorly in single-task or single-target transfer scenarios compared to multi-task learning?

[1] Zhao, Zirui, Wee Sun Lee, and David Hsu. "Large language models as commonsense knowledge for large-scale task planning." Advances in Neural Information Processing Systems 36 (2024).

• The motivation is not clear. The point that an IRL algorithms (AIRL, GAIL) can fail to recover the true reward function is highlighted multiple times. However, it depends on the formalism as described in [1], where methods like GAIL is a primal approach and the learned reward at convergence might not align with the true reward, whereas solving the dual form of IRL where the reward function is learned with no-regret approach, the method should recover a meaningful reward function. So, this might explain that IRL methods when trained in Dual form can recover the true reward function and AIRL is not able to learn it.
• The paper is not well written and hard to follow, especially the experiments section.
• According to the paper, the IRL methods do not learn good reward functions. It is not clear why the proposed method still uses this formalism for the scenario with unknown task rewards (Section 4.4).