How Can LLM Guide RL? A Value-Based Approach

While the paper introduces an interesting concept, it requires significant improvements, particularly in experimental design, explanation of baselines, and addressing concerns about practicality before it can be considered ready for publication.

It is currently unclear if a fair comparison was made to baselines, how it would perform without RL, how it would perform with a sub-optimal LLM policy, how domain-specific it is, how practical the code production and human review is relative to other methods, and whether it could be applied in other environments.

Further minor comments:

• The title should probably be "How can LLMs guide..." instead of "How can LLM guide"
• The paper mentions several times that (e.g. line 49) that "RL agents require huge amounts of random interactions", which comes across as the authors assume that any RL agent uses random action selection for exploration, even though there are much more sophisticated exploration techniques in the literature that do not explore just randomly.
• In line 99 you have the term $V^t_{h+1}(s')$ where $t$ was never defined in the text so I think this is an error you'd like to be aware of.
• In line 108 "the LLM policy is hard to be optimal" should be rephrased.
• Line 110 "priorin" spelling error
• Algorithm 1: I would suggest including where the LLM contributes to the algorithm as it is your main contribution, even if it is with a comment in line 3 of the algorithm.
• It would be nice to have some more details on how exactly you are learning a transition model, this was not very clear to me from the paper.
• In order to measure how optimal the LLM policy is in the given environment, it may be helpful to execute a pure LLM policy for one episode. Based on the accumulated reward, a weighting of the regularization term could be determined.

Following the weaknesses,

• Can authors provide more practival motivations? Besides, it seems that the provided algorithm highly relies on value-based RL algorithms, does such insight also make sense when leaverage policy-based methods such as PPO?
• What the techinical contribution/hardness in theoretical analysis?

Also, here's some additional question

• At the end of theoretical analysis, it is claimed that ``the core objective of our theoretical analysis is to justify the use of log-probability as regularization", but SLINVIT uses $\pi_{\rm LLM}$ rather thatn $\log\pi_{\rm LLM}$. Why there's a mismatch? Also, could author provide more elaborations on the connection between KL regularization and practical sum of log probabilities?
• (Minor) What's the difference between notations $\mathbb{P}_{\rm LLM}$ and $\pi_{\rm LLM}$, are these the same?
• (Minor) Is the $a$ missing in $\bar{Q}_h^t(s,a)$ for (3.4)?

1. Where is Eq. 3.1 taken from? It is presented as a specific choice of the author but it is surely defined in a different theoretical context.
2. I assume that the LINVIT is the theoretical motivation of SLINVIT, however it cannot act as so if it's not empirically validated. Have you tried running LINVIT in a tabular reinforcement learning setting? How can we know if the methodology is sound?
3. The rule-based method requires a human review, how is this process handled? Do you have guidelines on how this should be done? What if the LLMs proposed a value function that looks wrong but it correctly compiles?

Refer to the weaknesses mentioned above.

Additionally, the RAP framework utilizes MCTS for Blocksworld tasks, also use MC sampling and "likelihood of action" as a reward, similar to the approach in Eq. 5.1 of this paper. Given these similarities, what is the primary reason for the substantial performance advantage of SLINVIT over RAP?