IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic

Dear reviewer,

Thanks a lot for appreciating our work !

Please find our responses to your comments below.

The experimental design is sound, however, it uses too few seeds (5). It would also be useful to compare the performance for different data set sizes (the appendix does contain experiments with a single expert trajectory, though).

We will increase the number of seeds. As can be appreciated in the appendix it seems that even with just 1 expert trajectory SOAR improves against the non exploratory counterparts.

Exploration strategies, also based on optimism, ... in sample efficiency.

We definitely agree with the reviewer. Indeed, we hope that the main takeaway from our submission is that optimistic exploration in imitation learning is beneficial. This fact does not seem to be very well known in the current IL literature.

Very recently, Moulin, … manuscript.

Thanks to the pointer to this reference we missed. We will add it in our revision

The main contribution … theme).

In our opinion, the 2 contributions nicely complement each other. Indeed, our goal was to find a method achieving these two desiderata simultaneously:

1. Guarantees in the tabular case.
2. Convincing empirical performance in continuous control experiments with minimum algorithm modifications ( that is the way the critics are trained).

The ensemble based exploration technique implemented in SOAR is one possible way to achieve this goal. Notice that, the tabular guarantees per se could also have been achieved with classic analysis leveraging exploration bonuses but it is very unclear how bonus based algorithms can be implemented with neural networks.

In Algorithm 1, line 11, it seems like $k$ should rather be $K$

Thanks for catching the typo ! We will fix it.

Line 372 refers to an "attainable range". How is this defined?

We will clarify this. By attainable range we mean the interval in which the state value function can take values. Since for all $s \in \mathcal{S}$, we have that $V(s) \in [0, (1-\gamma)^{-1}]$ we consider as attainable range the interval $[0, (1-\gamma)^{-1}]$.

Thanks again for the time spent reviewing our paper, we remain available for further clarification if needed.

Best, Authors

Dear reviewer,

Thanks a lot for appreciating our work !

We will make sure to capitalize the algorithm names in the figures.

Thanks again for the time spent reviewing our paper, we remain available for further clarification if needed.

Best, Authors

Dear reviewer,

thanks a lot for appreciating our work and for all the suggestions ! Please find below the responses to your questions.

Have you thought ... overhead. We agree with the reviewer! We think that interesting next steps are applying the exploration technique based on multiple critics in RL and RLHF. For this paper, we feel that it would be better to keep the message more coincise to imitation learning. We can definitely mention in the future directions this point.

Why are you adding 2 ... A larger additional weight in the denominator is needed for technical reasons. In particular, to invoke Cassel et al. (2024, Lemma 3) in the proof of Lemma B.1 which in turns applies Corollary 1 (b) in [1]. Intuitively, adding $2$ rather than $1$ in the denominator is needed to ensure that the estimators are more optimistically biased for a small number of visits.

[1] Wiklund 2023, Another look at binomial and related distributions exceeding values close to their centre

Given you're running ... interesting.

Thanks for the suggestion. It would be nice to see if optimistic gradient descent + optimistic exploration comes with any benefit.

We speculate that proving last iterate convergence using optimistic updates for both reward and cost should be possible but it could deteriorate the sample complexity. An alternative approach to get last iterate could be to strongly regularize the problem as in [2] but this comes at the cost of worst rates in $K$.

In the attempt we did to prove last iterate convergence with optimistic gradient descent ascent the problem is controlling the expected bonus under the last iterate distribution. Indeed, the technique we use to bound the average bonuses in Lemma C.2 does not seem helpful towards this goal.

[2] Müller et al. Truly No-Regret Learning in Constrained MDPs

If it's ... SOAR has no memory overhead over standard SAC.

Thanks for the suggestion ! We think that at the practical level this uncertainty quantification technique can be used effectively. We are more hesitant in saying whether guarantees can be proven about this technique. We will definitely add a discussion about this technique.

In Remark 3.1, when you ... clarify this.

The reviewer is right. We meant a single pair of critics in Remark 3.1 and we will clarify this in our revision. Thanks a lot for noticing this !

Response to the section "Other comments and suggestions"

Thanks for pointing out the missing references, for suggesting how to improve the abstract and the overall writing (especially for Section 3.1 and for the presentation of the algorithm) ! We are happy to implement these changes and include a discussion about the missing references in our revision.

Finally, thanks for suggesting AdRIL, HyPe and SMILING as possible algorithms to be boosted with SOAR. During the rebuttal time, we run a SOAR boosted version of HyPe and we noticed that it improves over the base version of Hype in Ant, Humanoid and Hopper. The results are available here https://imgur.com/a/2Zse8xl

Thanks again for the time spent reviewing our paper, we remain available for further clarification if needed.

Best, Authors

Dear reviewer,

Thanks a lot for appreciating our work and your careful review!

Please find the answers to your questions/remarks in the following.

..experiments using explicitly exploration-critical tasks would strengthen the central claims of the paper.

In order to assess the exploration capability of our algorithm we consider state only imitation learning in an hard tabular environment which is usually used to demonstrate lower bounds. We refer to [1] section 4 for a detailed description of the environment. We make it slightly harder considering 20 actions (instead of 2 actions as in [1]). Among the actions available to the learner there is one action $a^\star$ that has slightly higher probability to maintain the learner in the state visited more often by the expert. Since, we are in imitation learning from states only setting, the learner can not observe such action $a^\star$ therefore it has to explore efficiently the action set to estimate $a^\star$.

We use this environment for this experiment https://imgur.com/a/HI3LOeq. The results show that exploration is necessary in this environment. Indeed, it is evident that for $L=1$, that is when the algorithm does not have an exploration mechanism, the learner fails at achieving expert performance while the learner succedes when multiple critics are added. ( We will comment on the choice of $L$ later in our response)

Regarding continuous control experiments, we are not really sure about which environment could be used to assess the exploration capability of an algorithm. Does the reviewer have some environments in mind ?

[1] Osband & Van Roy 2016, On Lower Bounds for Regret in Reinforcement Learning

There appears to be a gap between theoretical expectations and practical observations...

In figure 4 increasing the number of critics does not improve the performance further but this is not in contradiction with the theory. Indeed, notice that in Theorem 3.4 $K$ denotes the number of iterations while the number of critics is denoted by $L$. The performance of the algorithm is not expected to improve in a monotone way at the increase of $L$ because of the following tradeoff:

1. $L$ should be large enough to ensure high enough probability of optimism according to Corollary 4.10 ( see also Lemma B.1)
2. $L$ should be small enough to guarantee a tight bound on $\delta^k$ according to Lemma 4.12 where the number of critics L appears at the numerator.

Therefore, it is expected from the theoretical study that an excessively large value of $L$ leads to worst sample complexity guarantees. Ideally, $L$ should be set equal to the smallest value guaranteeing optimism with probability $1-\delta$. This is the reasoning behind setting $L = 7/2\log(2 SAK / \delta)$ in our paper. In order to verify the theory, we add an ablation on the number of $Q$ estimators ( going up to 100 as suggested) in the tabular setting which verifies that there is indeed a tradeoff in the choice of $L$. The image is visible via this anonymous link https://imgur.com/a/HI3LOeq

Question 1. From the theoretical analysis presented here the two aggregation rules are equivalent. In practice we think that mean+std offers more flexibility in tuning the clipping parameter. Therefore, while equivalent in the worst case the std based aggregation rule seems to achieve better instance dependent results. It would be interesting to confirm this observation theoretically in future work.

Question 2.

We did not try to increase the number of critics above 10 in the continuous states and actions experiments with neural networks. We think this should make the algorithm’s computational cost excessively high compared to a single critic implementation. In the tabular environment considered here https://imgur.com/a/HI3LOeq we try a larger number of critics and see that there is a tradeoff in the choice of $L$ ( please see explanation above).

Thanks again for the time spent reviewing our paper, we remain available for further clarification if needed.

Best, Authors