Policy Learning For Video Streaming

1. The main weakness seems to be the lack of novelty. Of course, DQN has not been used before, however, RL approach has been used (Pensieve). The paper did not explain why their approach has a better performance compared to Pensieve. The reward structure is the same, hence, the reviewer is wondering what are the advantages.

2. Even though the performance seems to be good, but it is not improving by a lot. While it is fine in general, but, given the fact that the methodology is not novel and the improvement is not a lot, my recommendation will be towards rejection.

3. The proposed algorithm has not been tested for a wide range of dataset in particular for 5G dataset.

The motivation and the advantage of the proposed algorithm compared to existing reinforcement learning based approaches is not so clear to me. The authors stated that: “MPC optimization occurs online, and so setting the look-ahead window to be too far into the future can be associated with prohibitively expensive computational burden and time. “ This statement doesn't align with my perspective. To illustrate, in Pensieve (Sigcomm 2017), the look-ahead window comprises only a single predicted or estimated input to the RL agent, which serves as an approximation for solving the optimization problem. Furthermore, in practical terms, it's essential to keep the window size relatively small. As the window size increases, the accuracy of predictions tends to decrease.

Moreover, they stated that: “In this study, we revisit ABR policy learning. We present a simple ABR scheme, called “Wolfi”2. Wolfi applies deep-Q-learning to the exact same inputs required for MPC’s online optimization process, thus replacing MPC optimization with policy learning. “ Yet, this is precisely the approach adopted in Pensieve (Sigcomm 2017): approximating the solution to the online MPC optimization problem using a reinforcement learning agent. The question then arises, how does this current work distinguish itself from the research conducted five years ago?

The authors have also put forth the following claim: “Solve an exact optimization problem to determine the sequence of bitrates that maximizes a QoE objective (of the above form) with respect to the predicted download times.” However, I contend that, in most cases, it is not necessary to find a solution to the precise optimization problem. Numerous algorithms have tackled simplified variants of the original problem or introduced heuristics that have demonstrated strong performance. In fact, in FastScan (published in IEEE Transactions in Circuits and Systems for Video Technology, 2019), a low-complexity algorithm was introduced to solve the exact problem. The formulation they utilized is identical to the one employed in this paper.

The issue outlined in the context of Pensieve, and I am quoting the lines here:

“However, while Mao et al. (2017) demonstrated better performance than previous methods, subsequent large- scale empirical investigation demonstrated that, when unleashed in the real world, Pensieve exhibits poor QoE (Yan et al.). This can be attributed to the gap between Pensieve’s training environment (a simulator) and the complexity of real-world traffic conditions, as well as to DRL’s notoriously brittle optimization process.” I believe this issue is not exclusive to Pensieve itself; rather, it applies to any algorithm. In other words, any algorithm lacking training on a comprehensive dataset that accurately represents real-world scenarios will likely underperform. This is not a problem specific to Pensieve.

• Evaluation of the streaming utilises the puffer testbed. It is mentioned that a server serves multiple clients over varying conditions using an emulator. No details are mentioned how many clients are setup. Interaction of multiple streaming clients over the network and its effects over rebuffering events, QoE, (particularly considering tcp window scaling effects in real world) is not studied.

• SSIM is not the strongest quality indicator for QoE (see question below)

The level of contribution, particularly on the ML side is very low. No new methods or algorithms are developed. Essentially, the main idea is that an RL approach (using a DQN variant) may be used to produce a similar performance as an MPC in the case of video ABR. The supervised learning part to generate some of the features (state information) used by the RL is obtained from a previous work. While this is a nice combination, the notion of RL as yielding a policy comparable (or superior) to MPC is well known. This paper should probably be submitted to a networking venue such as those where Mao-2017 and Yan-2020 appeared.