Generalized Policy Iteration using Tensor Approximation for Hybrid Control

We greatly appreciate your valuable feedback and suggestions and include them in the paper. Here, we offer responses to specific questions and concerns that you have thoughtfully presented.

Although the example on the real robot is very impressive, the examples in simulation are less so. Four-dimensional state space is not that high, barely beyond what can be represented with a look-up table on modern computers. (10^8 cells will take around 400MB of FP numbers.) The authors clearly state that their algorithm is not meant to approximate value functions on very high-dimensional state spaces, such as images, but most robotics applications on 6 degree of freedom robots have 12-dimensional state space, so this is perhaps the dimensionality of highest interest.

How well will the algorithm perform on a somewhat higher dimensional problem, for example, a 6 or 12 dimensions?

Indeed the dimension of the state-space in the benchmark simulation examples for hybrid control is rather low (4D). However, the benchmark simulation problems are representative of high-dimensional action space problems and it is quite large (up to 32D). In general, in our approach what matters the most is the dimension of the state-action space. We believe the crucial aspect is that TTPI can handle high-dimensional action space which is difficult for existing ADP algorithms.

In the software library provided, we have included the regulation problem of the point-mass system with obstacles. We have tested it for various dimensionality as it easily forms a testbed for high-dimensional state space. Here the state space is $2m$ where $m$ is the dimension of the action space (acceleration of the point-mass which is the same as the dimension of the environment it lives in). We have tested it for $m = 2, 3, 4, 5$ (so state space of size $4D$ to $10D$) and observed its performance matches the classical algorithms. So, we believe TTPI should be able to handle a relatively larger state space dimension of interest in robotics problems. We will include more examples of robotics problems in the future release of the software.

I am satisfied with the authors' clarification about state space dimensionality - the joint dimensionality of the state-action space is indeed high enough to make it impossible to use a tabular representation, and the benefit of using the tensor-train representation are clear.

We greatly appreciate your valuable feedback and suggestions. We also appreciate your suggestions about other discretization procedures that we could employ. Here, we offer responses to specific questions that you have thoughtfully presented.

did you use ideas from opitma_tt (paper Chertkov et al. (2022)) which utilize top-k scheme and thus more accurate?

As mentioned in the paper, we have included the ideas from optima_tt (which is the deterministic version of TTGO). The main advantage is that the policy iteration is stable due to the deterministic nature of the policy.

what are the typical TT-ranks for double pendulum swing-up in the case of small when the model does work?

For the cartpole swingup and double pendulum experiments, we had to use a time step lower than 1 ms. This resulted in a TT rank of the advantage function between 50 to 100 over the policy iteration stages while the value function had a rank was about 50.

We greatly appreciate your valuable feedback and suggestions. Please find below our response to your question.

Tensor operations, especially in high-dimensional spaces, can sometimes introduce numerical instability. How does the TTPI algorithm ensure numerical stability, especially in long-horizon problems?

In our algorithm, we specify the accuracy of approximation of the value function and advantage function in TT-cross. In our experience, an accuracy of 0.001 is usually enough to avoid any instability. The numerical inaccuracy in the forward simulation is another challenge. In general, TTPI can be robust to this as it is an ADP algorithm (so it approximates the value functions almost everywhere in the state space) and our algorithm does not require long rollouts in the Bellman update (we used rollout of only one step (about 1ms) in the algorithm).

Regarding hand-coding the dynamics, TTPI can work with a simulator as it only requires samples of the tuple (state, action, next-state, reward) queried by TT-cross for modeling the state-value and advantage function. The main requirement from the simulator is that we can query this tuple from an arbitrary state-action pair specified by TT-cross in batch form. However, as modern simulators such as NVIDIA Isaac Gym are primarily geared for RL, it still requires some effort in such customization. We are currently investigating this.

We greatly appreciate your valuable feedback and suggestions. Here, we offer responses to specific questions and concerns that you have thoughtfully presented.

My main concern of the paper is the experiment, which is quite limited and there are many remaining questions. Particularly, there are many hyperparameters, it is expected to have ablation study of showing the performance of the method versus the hyperparameters, such as accuracy of TT representation, the max TT rank, discretization resolutions of the state-action spaces....

As a reader to implement the algorithm, I wanted to see a straightforward illustration between the running time of the algorithm and dimensions of the state/action spaces of the system, or directly give an overall complexity of the algorithm for one policy iteration.

The hyperparameters in TTPI are intuitive and we have observed them to be robust to the changes in the hyperparameters. The four main hyper-parameters are (1) the upper bound on the TT rank $r_{max}$ of the state-value function and the advantage function (2) the accuracy of tt-approximation $\epsilon$ (3) the number of samples used in TTGO in computing the policy (4) the number discretization of the state and action space. We have maintained the same choice of hyperparameters in all the experiments across different robotic tasks, so believe it is quite robust.

In general, due to the properties of TT modeling, the computation involved and memory required grows linearly with respect to the number discretization and the dimensionality of the state-action space, and quadratically w.r.t the rank.

In our experiments, we kept the maximum rank to a large enough value (100 but almost all the experiments never reached the maximum rank in TT-cross modeling over the policy iteration procedure). The accuracy of approximation $\epsilon=0.001$ was sufficient and decreasing epsilon will improve the performance slightly but at a higher computational cost. The number of discretizations of each variable (state or action) is something that depends on the problem at hand and it is fairly straightforward in robotics problems. For example, if it is a joint angle ranging from $(-\pi, \pi)$, a discretization of about 50 to 100 suffices as we employ interpolation for continuous variables. For (3), the higher the number of samples used, the more optimal the policy computation at the cost of linear growth in computational cost and the memory requirement. We set it in the range of 10 to 100.

Since the majority of the paper is about the background of TT (previous work), I think the main contribution, which is TTPI algorithm, needs more explanation, including its parallel implementation, which seems a key for applicability of the algorithm. For example, why we need a TT-round operation for the value function in line 8?

The reason for TT-round operation is that the TT representation obtained by TT-cross need not be optimal in terms of the number of parameters used and the same tensor in TT format can be represented more compactly with the TT-round operation.

We will add more information about the implementation details in the paper.

Since in the introduction the authors mentioned RL, I think it would be interesting to discuss the potential of the methods into "model-free" settings.

We believe our approach can be used for RL and policy learning in general. TTPI can be extended to RL in the following two ways:

(1) Model-free RL: In this setting, TT as used in TTPI could be used as the function approximator. The difference would be instead of using TT-cross to find the TT representation of the value function or advantage function, we can use gradient-based techniques to find the TT approximation to the value and advantage function (or directly the Q function which will result in Q learning). The policy computation from the advantage model (or Q model) is done as in TTPI. We believe this idea has a huge potential in policy learning in general.

(2) Model-based RL: In this setting, a model of the system is learned using NN or any other function approximator. Then, we can apply the TTPI algorithm using the learned model.