Generative Reinforcement Learning with Transformers

I do not think that the paper has suffecient contributions for ICLR. There are not foundemental contributions on the theory side. The only contribution appears to be on the method side. The authors seem to use foundation models to approximate the state distribution. I am not sure how novel or impactful that is.

On the practical side, it appears that the authors have made quite some effort in the experiments. However, they do not compare their method/results with any similar methods which makes is hard to assess whether there is any improvements compared to the state of the art. Also, for a paper that's such heavilly application-oriented, I'd expect to see clean, well-documented source code in the supplementary material.

I have several open questions (please see below).

Additionally, the experimental results are interesting, but not convincing w.r.t the strength of the generative model and in particular fine-tuning only seems to provide very marginal gains.

Furthermore, the authors claim that "Our results support the Value Function Hypothesis, in the sense that the generative value function approximation techniques produce far better policies than behavioral cloning". I don't think BC is the right comparison as it does not make use of the reward at all. In my opinion, a more suitable comparison would e.g. be the decision transformer which still makes use of rewards, but without using a value function as an intermediate step.

• The paper spends a lot of space discussing basics (more than 4 pages), then a tiny section on the methods, which is supposed to be the main part. In particular, the section on "Objective Functions" is unclear. The authors have used statements like "X is better than Y" at times with little reasoning. I would strongly suggest to shorten introduction and basics, and add discussions to support such statements, either formal or at least through toy examples with reasonable explanations.

• The chess player agents using supervised learning are still quite important benchmarks to compare against (the authors named them in the second last paragraph of the related work section). Indeed, this is the only way to prove RL is a requirement and rebuttal recent voices that supervised learning reins supreme. Once you learn the value function, its corresponding updated policy (greedy or Boltzmann) can be compared against such agents. Or at the very least, their Elo scores can be compared.

1. There is lack of summary of distributional RL in the related work.
2. Page 4, MDP M = (S, A, \mu), \mu is the transition probability kernel in the CNC paper, should use \mathcal{P} here.
3. The paper focuses on comparing discriminative and generative decomposition of \xi (z | s, a) for estimating the value function. Yet it's not clear how well the value-based approach compared to the decision transformer approach which directly updates the policy. I suggest adding these experiments for a more comprehensive comparison.
4. The major limitation of this paper is, although it provides some evidence and insights, showing the potential of estimating value function using RL, it's not conclusive whether the value function hypothesis is true. The behavior cloning baseline is too weak to reach a conclusion. Also, the generative approach has not shown advantage over discriminative decomposition, even that it was finetuned on a strong foundation model. This work brings more questions than answers and I do wish there are more answers resolved in this paper.

• The paper feels like a proof-of-concept:
  • The proposed method does not conclusively answer the question asked in the paper (about the Value Function Hypothesis).
  • The proposed method does not beat a baseline (a superiority of discriminative methods was observed).
• Section 3
  • The paragraph "connecting value function to the logarithmic loss" is unclear.
    • The promised connection is not very transparent.
    • A statement claiming that a set of augmented states converges to a stationary distribution is confusing (it seems like a "type mismatch" problem).
  • Section 3, paragraph "discriminative and generative decompositions"
    • The decomposition approach is confusing
      • The main components are $\xi(s|a,z)$ and $\xi(z|a)$. Both are rather unorthodox when it comes to RL: the former requires the knowledge of future returns ($z$), and the latter gives the distribution over returns only given an action (without knowledge of the state $s$).
      • How are such objects used for inference? A pseudo-code could improve the clarity here.
    • Discussion following eq 3 is unclear (e.g., why is $\hat{\xi}(s|a,z)$ good and $\hat{\xi}(z|s,a)$ bad?)
    • There should be an assumption that $|\mathcal Z|<\infty$.
• Experiments:
  • Section 5.1:
    • The evaluation protocol is confusing:
      • Is the computation of ELO metric and early stopping rule described in the "main result" standard? Is there a citation that one can refer to?
      • In the last sentence of paragraph "dataset", it is written that models are evaluated on ~24K boards. Where is it reflected in the paper (Table 1 says about 800 games, roughly 50 for each pair of agents, or computing metrics on 28K boards)
    • Table 1 implies that the generative approach has ELO roughly smaller than 100 when compared to the discriminative approach. According to the table's caption, the latter is twice as likely to win. This seems to conflict with the statement that "the two main decompositions are close in performance".
    • The results of the ablation part are not very informative (what is the main conclusion? are the conclusions for smaller models also valid for larger ones?). It seems more like content for the Appendix.
  • Section 5.2 (gridworlds).
    • Experiments do not seem very informative, showing that it is impossible to discriminate between the methods significantly.
    • The (online) training protocol could be better explained (e.g., in the form of a pseudo-code).
    • It is curious that using a constant uniform return model does not impact the learning dynamics. It would be interesting to see the results in similar environments where this is not the case.
  • Other:
    • The paper often first describes what did not work and then what was done. For example, eq (3) or the second paragraph in Section 4 "return-action conditioning". While the information about what did not work is very valuable in machine learning, it could be gathered in a separate place and could possibly be put in the Appendix.