Finally Rank-Breaking Conquers MNL Bandits: Optimal and Efficient Algorithms for MNL Assortment

Thanks for your careful reading and insightful comments. We address your questions below:

W1. Main Motivation - In what kind of applications would this new AOA model (where NC need not be strongest) be helpful?

-- The likelihood of users' preference of selecting the "No Choice" (NC) item widely varies over application. You are absolutely correct that in e-commerce applications, online ad placement or even during web-search, no-choice might be a popular (strongest) choice, however it might not always be true. For example:

• In recommender systems like YouTube, Spotify, Netflix or even Yahoo News, users typically make choices as they actually wanted to consume a video, new article or song, etc.
• Similarly, in shopping recommendations like flights (Expedia or Google flights), hotel (Booking.com), restaurants (Grubhub), or Google Maps recommendations, NC is unlikely.
• While voting or in crowdsourcing platforms where a business platform/product designer is collecting feedback on a bunch of items, no choice might not be the most preferred option.- The smart text recommendations we receive on our phone (while typing or replying to text), no-choice is often not the most preferred option.
• In fact, in some applications, NC might not even be an available (feasible) option, e.g., while interacting with ChatGPT/ Gemini, the language model often requests the users to definitively select one outcome in order to proceed with the thread; the language models must receive a preference feedback from the user in order to continue with the thread. Similarly, in robotics applications, training of autonomous vehicles, or more generally in any preference-based RL (a.k.a. PbRL) applications, the teacher/ demonstrator/ human feedback provider must choose an option out of the multiple options (or RL trajectories) towards training the RL policy. Here, also, selecting NC is not an option. 

We already added these examples in the updated draft in the openreview portal. Please advise if you suggest any further changes.

In conclusion, thus the strength of the NC item very much relies on the underlying application, and indeed, assuming NC to be the strongest item becomes a limiting (restrictive) assumption in such situations. It is also worth noting that this is also the reason the original MNL-UCB paper (Agrawal et al'2019) tried to relax the assumption of \theta_0 = \theta_\max but due to the limitation of their algorithmic techniques, their regret bound stands vacuous in this regime. We explained this in more detail in the context of your next question. 

W2. If the rank-breaking algorithm is applied to a setting where the no-choice item does dominate, how does the algorithm compare to MNL-bandits?

Thanks for the question. One of the main goals of this work was to lift the restrictive assumption of ``strongest NC item" (i.e., $\theta_{\max} = \theta_0 = 1$) from MNL-bandits. Please note Table-1 compares our relative performance with that of MNL-bandits (Agrawal et al'19). Indeed the regret bound of  Agrawal et al'19 is optimal when NC is strongest. However, our focus area is the regime of small $\theta_0$. A key motivation of our contribution is that our algorithm (Alg 2, AOA-RBPL-Adative) works in the $\theta_0 \to 0$ regime (equivalently $\theta_{\max} \to \infty$) (see Thm 5), where we achieved large improved regret bound of just $\tilde O(K \sqrt {T})$ compared to the vacuous $\infty$ regret bound of Agrawal et al (2019) as $\theta_{\max} \to \infty$. Here lies our key improvement, which was possible owing to the rank-breaking-based MNL parameter estimation technique we posed in its paper. Note that despite the attempt of Agrawal et al. '2019 to lift the assumption of $\theta_0 = 1 = \theta_{\max}$ (please check their Sec 6.2), they only managed to derive a vacuous bound which shoots up to $\infty$ as $\theta_{\max} \to \infty$ (equivalently $\theta_0 \to 0$). This trend is also validated by our experiments (please see Fig 1 and 2, Sec 5).

-- The remaining questions are continued in the thread below --

We appreciate the reviewer's feedback and recognize the valuable insights provided. Below, we address the specific concerns raised:

While we acknowledge that the novelty of contributions is subjective, we respectfully emphasize the significance of our work:

• Practicality of non-repetition: Our work does not dismiss the potential advantages of repeatability in user interactions; instead, it addresses scenarios where showing the same set repeatedly can lead to dissatisfaction or disengagement. For example, in dynamic contexts like ad placements or content recommendations, repeated exposure may annoy users. Our algorithms offer flexibility to cater to diverse application settings by avoiding this limitation in a scalable and regret-optimal manner. Our primary critique pertains to the methodological requirement for exact repetition of selections in earlier works, which may not be feasible in many real-world systems. Our approach eliminates this dependency while achieving optimal regret bounds.
• Our critique of the reliance on a default "No-Choice" item stems from its impracticality in scenarios where the default option is rarely selected. We offer a model that generalizes this assumption without sacrificing regret performance. Our experiments (Sec. 5) demonstrate that our methods outperform existing baselines in scenarios with low NC parameter dominance, illustrating their real-world relevance.

We thank the reviewer for suggesting the use of the parameter estimation technique for MNL from Agarwal et al., ICML 2018. While we agree that this approach might achieve sub-linear regret and merits exploration, we believe it is not straightforward that it would achieve the same regret bounds. Specifically:

1. The results in Agarwal et al. are expressed in terms of the total variation norm rather than the $\ell_\infty$ norm, potentially yielding a suboptimal factor of $K$ (the number of items).
2. Their estimation bounds depend on the spectral gap of the random walk $\mu(P)$, which can scale as $1/(m \max_{i,j} (w_i/w_j)^2)$ (see Corollary 1). This introduces undesirable dependencies not only on $m$ but also on $\max_{i,j} (w_i/w_j)^2$, which our approach explicitly aims to avoid.
3. The suggestion to add a reference arm is unclear—how should such a reference arm be chosen in practice? In contrast, our procedure uses a reference arm only in the analysis, not in the algorithm itself.
4. Finally, their method would still suffer from the limitation we address: requiring repeated display of the same set in long instances, which we aim to avoid.

Please let us know if you have any additional questions. We will be happy to clarify.

Thanks for your comments. We would like to point out a few things that you might have missed, which might perhaps cause confusion.

-(A) Please note our main result is Thm. 5 and not Thm. 3: Firstly, there is a great misunderstanding on the reviewer's part to identify our main result. Please note, as we also clarified in our list of contributions (Sec 1) and in the Expts (Sec 5), our main result is Thm. 5 and not Thm. 3. Thus, as opposed to what you claimed, our result (Thm 5) is completely independent of the ratio $\theta_{\max}/\theta_0$ -- in fact, this is the key contribution of our work over MNL-Bandits (Agrawal et al'2019) which already establishes a regret bound of $\sqrt{\theta_{\max}/\theta_0 KT}$ regret bound, which is obviously vacuous in the regime of $\theta_{\max } \to infty$ keeping $\theta_0 = 1$ (or equivalently the regime where $\theta_{\max} >> \theta_0$). We clearly mentioned this advantage in several places, including Rem 1, Expts 1 (Averaged Regret with weak NC $\theta_{\max}/\theta_0 >> 1$), as well as in our list of contributions. Thus, please note our goal was to actually derive a $\theta_{\max}$ independent result, and we successfully did so with our Adaptive Pivot Selection algorithm in Sec 4, which provably runs in $\tilde O(K\sqrt T)$ regret (Thm 5). So your claim that

``your rank breaking algorithm also has a rather suboptimal dependence on this ratio $\theta_{\max}/\theta_0$ ... so its essentially the same as having a $w_{\max}/w_{\max}$ type dependence" is incorrect and stands vacuous.

-(B) Our Algorithmic novelty lies far ahead of just the Rank Breaking (RB) idea: Secondly, note that our novelty not only lies in rank-breaking based estimation. There are several novelties behind our algorithmic idea that makes it regret optimal as well as computationally efficient. The novelty not only lies in our algorithmic ideas but also regret analysis which is equally interesting and involves non-trivial mathematical rigor. We explain this fully in the thread Revisiting the key Ideas of our main algorithm (Adaptive AOA-RBPL, Sec 4) and its regret Analysis above. Further, owing to this novelty, it is also easily extendable to general preference models, including RUM models as explained in Rem 2, unlike the prior works or even the Spectral ranking method you described above.

-(C) Hand wavy claims vs our Concrete proofs with solid empirical evidence: Finally, with all respect, we firmly disagree with the reviewer that our result (Thm 5) can be obtained straighforwardly from the Spectral Ranking (Agarwal et al'2018) paper. If it were so easy, we should have been able to derive it from a straightforward extension of their result, which is clearly not the case as coming out from this discussion conversation.

Here are the reasons why it is not straightforward. Firstly note our algorithm (Sec 4, Thm 5) is already independent of any such ratio term such as $w_{\max}/w_{min}$ as opposed to what you proposed for the derivation. Even if we ignore that for a moment and wish to derive a bound scaling with $\max(w_i,d_i)/\min(w_i,d_i)$, it is absolutely not clear how *``such dependency can potentially give an improved dependence on the ratio in the regret by setting $d_i \propto 1/w_i$''. Firstly, (i) $w_i$s are initially unknown to the learner, so a cost of estimation has to appear, which will contribute some multiplicative factor in the final regret guarantee. Secondly, (ii) if $w_{\max}/w_{\min}$ is small, this would yield a very high choice for the $d_i$ as it is set $\propto 1/w_i$ which would first increase the regret because of the dependence on the $d_i$. Finally, (iii) the computational complexity will shoot up, which will now need to construct many comparison sets, which will be a huge disadvantage compared to the minuscule computational overhead of our proposed algorithm in Sec 4. And (iv) all these disadvantages exist even if we ignore the fact that our results are already independent of any ratio terms like $\theta_{\max}/\theta_0$ which your proposed algorithm will not be.

In summary, as theoreticians, we understand certain concepts might appear simple with informal or hand-wavy arguments without concrete derivations and proofs. However, the actual analysis often proves to be far complex and nuanced when formal algorithms and rigorous proofs are developed. If this was indeed so simple, it would likely have already been addressed and published, given the strong and practical motivation of our work (to generalize the results of MNL-Bandits by relaxing the strength of the NC item)—as admitted by the reviewer themself. We therefore respectfully urge the reviewer to kindly not undermine and disregard the novelty of our algorithm, analysis, and experimental studies in favor of a (hand-wavy) idea that has significant limitations, as we have outlined above.

Pleasse let us know if you have any additional questions, would be happy to clarify.

I thank the authors for their response. I disagree with the authors conclusions on a few points. Regarding their comments about repetition vs non-repetition, I still maintain that an argument can be made one way or another, and both would remain valid unless one provides actual evidence supporting one over the other. Moreover, I still don't see repetition being a "weakness" since again, this online learning process isn't being done with a single user, rather a whole population, in which case repetition vs non-repetition becomes a rather meaningless discussion. I agree with the authors point about generalization of the no-choice assumption.

Regarding their comments about using the MNL parameter estimation guarantees of Agarwal et al., ICML 2018, there are few potential misunderstandings on behalf of the authors. A lot of the pessimistic constants in their analysis appear due to the fact that they operate in the passive data regime, where one does not have control over the item sets that are compared. This problem on the other hand is in the active learning setting, where you can choose what item sets are compared, and as a consequence, have direct control over the topology of the underlying comparison graph, and you can choose sets such that the underlying comparison graph has a constant spectral gap, which is why I said using their method might give you much better dependence on problem constants. If you actually look at the proof of Lemma 7, where the spectral gap is bounded, the dependence is actually much weaker before it gets crudely bounded by the squared terms you mention in your response since there is no control over what the underlying graph may look like, hence they have to assume the worst possible case. Your rank breaking algorithm also has a rather suboptimal dependence on this ratio ($\theta_{\max}^{3/2}$ and $\theta_0$ is assumed to be 1 so its essentially the same as having a $w_{\max}/w_{\min}$) type dependence. Their algorithm actually has a dependence on something like $\max (w_id_i)/\min(w_id_i)$, which can give a potentially improved dependence on this ratio since $d_i$ is essentially the number of times item $i$ is compared, so in theory, if you scale $d_i \approx 1/w_i$, i.e. compare low weight items more than high weight items (which you can somewhat identify since low weight items will rarely win, and of course while trying to keep regret low) which again is possible to do since you get to choose what sets are played and how often, this dependence may go away completely. Moreover, since you can choose which items to compare, you can avoid having large weight items in the same comparison set as low weight items using say empirical selection probabilities to create these comparison sets (ideally, each comparison set should have items with comparable weights, and if you have really low weight items, then they should be identifiable and hence discardable quickly since they will be rarely selected). This would again circumvent this $w_{\max}/w_{\min}$ dependence, and would rather depend on this ratio only constrained to the items in the same comparison set, which would be a substantially weaker dependence than the global max to min weight ratio. In all, I am not convinced that this result is not already implied by a straightforward modification of existing work, and I am inclined to maintaining my original score.

Q2. How does the regret result relate to the maximum of assortment size m? ... How to understand the impact of $m$ on the regret?

Thanks for the great question! The answer is the problem complexity does not depend on the subsetsize $m$! --- Although it could be counter-intuitive but this is a known phenomenon with this class of MNL bandit problems. To see this mathematically, note the regret bound derived in Chen and Wang' 2017 is already independent of $m$ as we explained above in Q1. Further proof of tightness comes from the $\tilde O(\sqrt{KT})$ matching upper bound results which establishes the tightness of the regret lower bound. Intuitively, this could be explained by the logic explained in the Remark 2 Saha & Gopalan'2019 (https://proceedings.mlr.press/v98/saha19a/saha19a.pdf), which explains that

• Surprisingly ... the problem complexity of identifying a near-optimal item from m-sized subsets does not reduce with m, implying that there is no reduction in hardness of learning from the pairwise comparisons case (m = 2): On one hand, one may expect to see improved performance as the number of items being simultaneously tested in each round is large ($m$). On the other hand, the performance could also worsen, since it is intuitively ‘harder’ for a good (near-optimal) item to win and show itself, in just a single winner draw of $m$ items against a large population of $m$ − 1 other competitors which increase the variance in the regret model and making the learning problem harder. The result, in a sense, formally establishes that the former advantage is nullified by the latter drawback making the final regret performances independent of the subsetsize $m$.

Also, when m = 1, the learner is supposed to play subsets $S_t$ of size $1$, which means they only get to select one item that competes with the NC item every round (note NC is never counted in the subsetsize $m$ as it is the default item present in every set). In this case again, the regret learning tradeoff remains the same and independent of $m$ following the same logic and intuition we provided above.

More interestingly, as a side note it is worth pointing out that, if you note our experiments in (3) Averaged Regret vs Length of the rank-ordered feedback ($k$), Sec 5, we show that, if we make the feedback model of Eqn (1) richer with $k$-length rank-ordered feedback, where at every round upon playing the $m$-sized subset $S_t$, the learner gets to see the top-$k$ draws ($k \leq m$) from the PL model without replacement, then the regret performance indeed improves with increasing $k$, as also intuitive. Our Figure 3 in Sec 5 justifies this tradeoff. This also corroborates the conclusion of Remark 6 of Saha & Gopalan'2019, which claims that how performance improves as the learner has access to larger length rank-ordered results compared to just the winner feedback in the PL model considered in Eq (1), which is equivalent to k=1. I hope that clarifies your question.

Thanks again for the questions; we will definitely add comments to clarify both justifications for Q1 and Q2.

Please let us know if you have any additional questions. We will be happy to clarify.

Thanks for your careful reading and insightful comments. We address your questions below:

W1. In Line 315, Optimality of the regret bound.

Thanks for the question. You are right that stochastic multiarmed bandits have $\Omega(\log T)$ regret, but please allow us to first explain the difference between an instance (gap) dependent and instance independent (worst case) regret guarantee. This would clarify the confusion with adversarial and stochastic regret bounds. The standard K-armed multiarmed bandits (MAB) have two types of regrets

• (i) $\Omega(\sum_{i =1 i \neq i^*}^K (\log T/\Delta_i + \Delta_i T))$ type instance (gap) dependent regret bound where $\Delta_i$ is the reward gap of the $i-th$ arm and $i^*$ is the the arm with the highest reward (the a.k.a. best arm).

• (ii) $\Omega(\sqrt{KT})$ worst case (instance independent) regret guarantees. The name is justified since this regret bound is independent of the problem-dependent parameters ($\Delta_i$s). Note one can derive this regret bound from the above (instance dependent) lower bound for the (worst case) choice of $\Delta_i = \sqrt{K/T}, \forall i \in [K] \setminus \{i^*\}$ which indeed maximizes the instance-dependent lower bound in (i).

The book Bandit Algorithms by Lattimore and Szepesvari (https://tor-lattimore.com/downloads/book/book.pdf) explains this very nicely in Chapter 13, 15 and 16. Please note both the above lower bounds apply to stochastic MAB problems, i.e. the underlying problem instance used towards proving the above lower bounds were stochastic K-armed MAB instances. In fact, even if you notice the lower bound derivation (see Thm 3) in Lai & Robbins (1985), as you pointed, the classical result essentially used a stochastic problem instance to derive the $\Omega(\sqrt{KT})$ lower bound.

It is also worth noting that this means in terms of the regret objective, the problem complexity of adversarial MAB problems are no-harder than their stochastic counterpart (for the worst-case problem instances). Intuitively, the reason being this phenomenon is because the classical K-armed MAB instance evaluates the regret of a learning algorithm against a fixed benchmark, even if the underlying problem instance is adversarial. Now since adversarial K-armed MAB is only a harder problem than stochastic K-armed MAB, the lower bound derived in (ii) is also a valid lower bound for adversarial K-armed MAB (Note the lower bound of (i) is ill-defined for adversarial MAB instances as there is no notion of gap $\Delta_i$s in those settings).

Now, it is important to note that our problem, AOA-PL, is more general than K-armed MAB, and thus above lower bounds for the K-armed MAB might not directly apply for the problem for a general subsetsize m. However, the regret optimality of our method follows from the lower bound derived in Chen and Wang' 2017 as we mentioned in Line 353. The lower bound derived in Chen and Wang 2017 is $\Omega(\sqrt{KT})$, and it is of the nature of instance-independent worst-case lower bound, as explained in (ii) above. But it is a valid lower bound for our problem instance, in fact a tight one. It is also worth noting that deriving an instance (gap) dependent lower bound is much harder (compared to stochastic K-MAB) for our AOA-PL problem since the notion of gaps are different for our problem due to the different feedback model and regret objective. It is an open area of research which will require new mathematical rigor. Nevertheless, we show the regret performance of our proposed methods is tight in terms of the worst-case instance-independent lower bound as known by Chen and Wang's 2017. I hope that clarifies your questions.

-- The next question is continued in the thread below --

Thanks for your careful reading and insightful comments. We address your questions below:

W1. The writing of this paper can be further improved. Section 3 and Section 4 are too technical and a little bit hard to read.

-- Thanks so much for the suggestion; we agree Sec 3 and 4 could have been less technical, we will either omit the proof sketches fully or shorten them significantly and add more intuitive/textual description of the math equations in the extra space. It will also improve the readability and flow of the text. Thanks.

W2. All the experiments in this paper are synthetic.

-- Thanks for your comment. Please note we are unaware of any open-sourced real dataset for the problem that can be used to validate our algorithms for these particular problem settings. One of the challenges being since it is impossible to know the ground truth parameters $(\boldsymbol{\theta})$ of the MNL model, it is hard to evaluate the regret performance of the algorithms. This is also the reason why most of the existing works only reported synthetic data experiments (Agrawal et al'17, Leslie & Grant'23). Although Agrawal et al'19 claim to have used the `UCI Car Evaluation Dataset', however, in reality, they synthetically modified the item features and assigned synthetic scores to the PL parameters through a linear score transformation (pls see Sec 7.3 of Agrawal et al'19) which ultimately only results into a synthetic data experiment only. Thus we are unable to include any real-world experiments due to the lack of any suitable open-sourced dataset available for the MNL-AOA setting.

Further, since the main focus of our work has been primarily theoretical, we conduct synthetic experiments to validate our main theoretical guarantees. However, if you are aware of any suitable dataset, we will be happy to report the comparative performance of the baselines on that dataset. Please let us know if you have any suggestions. Thanks for your feedback.

W3. [typo] for Arith50 environment.

-- Thanks so much for pointing this out. Indeed, it should have been $\theta_i = 1 - (i-1)\times 0.02, ~\forall i \in [50]$, which results in $\theta_i$s to be positive. We have already corrected this in the updated draft, please advise if you suggest any further changes. Thanks again.

Please let us know if you have any additional questions. We will be happy to clarify.

While Alg1 carves out the basic building block of our $\theta_{i,t}^{ucb}$, one caveat lies in its concentration rate which shrinks at the rate of $O(1/\sqrt{n_{i0,t}})$ --- as established is Lem7, $n_{i0,t}$ being the number of rank-broken pairs of $(i,0)$ till time $t$. Thus ideally, we would need $n_{i0,t}$ to grow fast for a fast and tight convergence of $\theta_{i,t}^{ucb}$ to $\theta_i$. However, as Lem2 reflects, this might not be true unless either $0$ (NC) or $i$ is a `strong item' with sufficiently large MNL parameters (comparable to $\theta_{\max}$). This is since, as Lem2 shows $n_{i0,t}$ (roughly) grows proportional to $\frac{(\theta_i + \theta_0)}{\theta_{\max}}$ which could be quite small if both Item-i and 0 (NC) happens to be a "weak-items" in terms of their MNL scores, i.e. $\max{\theta_i,\theta_0} << \theta_{\max}$.

If we think, this is also intuitive since, if both $\theta_i$ and $\theta_0 = 1$ are small compared to $\theta_{\max}$, chances are very low that either of them will be picked as a winner of any round $t$, even if $i \in S_t$ and as a result they will never be "rank-broken" against each other resulting in a very small value of $n_{i0,t}$, weaker UCB estimates $\theta_{i,0}^{ucb}$ and finally a weighted regret bound of $Reg_T^{wtd} = \tilde O(\sqrt{\theta_{\max} KT})$ (Thm3) which could be large when $\theta_{\max}$ is large!

Understanding the problem, we remedy this with Alg2 (AOA-RBPL-Adaptive) in Sec4, where we devised a smarter UCB estimate of $\boldsymbol{\theta}$, while still keeping the basic intuitions from Alg1 (AOA-RBPL) intact:

$\boldsymbol{\theta}t^{ucb}$ justification for Alg2: As explained above, we realized "pivoting" on the NC for estimating $\theta_i$ could be a bad idea, especially if $1 = \theta_0 << \theta{\max}$. Towards this we made the following (seemingly interesting) observations:

• (1) We first note that: $\theta_i = \frac{\theta_i}{\theta_0} = \frac{\theta_i}{\theta_j}\frac{\theta_j}{\theta_0}$ for any $j \in [K] \setminus {i}$.

• (2) Then drawing motivation from the UCB estimates of Alg1, we further set $\theta_{i,t}^{ucb} = \gamma_{ij,t}^{ucb}\gamma_{j0,t}^{ucb}$, where $\gamma_{ij,t}^{ucb} = \frac{p_{ij,t}^{ucb}}{(1 - p_{ij,t}^{ucb})_+}, ~\forall i,j \in [K]\cup{0}$ (display of Line428).

• (3) The hope is if we can find a "strong element j" and pivot our rank-broken pairwise estimates ($p_{ij,t}^{ucb}$s) around $j$ for all $i \in [K] \cup {0}$, hopefully the that will remedy the caveat of Alg1 as detailed above.

• (4) In order to find such a "strong pivot j" (such that $\theta_j$ is comparable to $\theta_{\max}$), we set a dynamic $j = \arg\min_{j \in [K] \cup{0}} \gamma_{ij,t}^{ucb} \gamma_{j0,t}^{ucb}$ for each item $i \in [K]$ and time $t$, which by definition happens to be a relatively stronger item than Item-$i$ and 0 (NC).

• (5) Further, similar to Lem1, we also find the rate of concentration of our new UCB estimates $\theta_{i,t}^{ucb}$ in Lem9 (see Appendix B.6), which, as intuitive, is shown to shrink at the rate of $O(\frac{1}{\sqrt{n_{i,j,t}n_{j0,t}}})$. The nice trick was to note that this yields a sharp concentration for $\theta_{i,t}^{ucb}$ owing to our clever choice of the dynamic pivot j as described in the point above -- this saves us from the caveat arose in Alg1 due to a poor choice of the static NC pivot (Item-0).

• (6) Finally the above tight and sharp concentration of $\theta_{i,t}^{ucb}$ in Lem9 ensures the final weighted regret of $Reg_T^{wtd} = \tilde O(\sqrt{\min{\theta_{\max},K} KT})$ (Thm5) --- Note this could be at most $\tilde O(K\sqrt T)$ even if $\theta_{\max} \to \infty$. Here lies the drastic improvement of our algorithm compared to prior works listed in Table 1 which either needs to assume $\theta_{\max} = \theta_0 = 1$ or their regret scales as $O(\sqrt{\theta_{\max}KT})$ leading to a trivial regret bound (as $\theta_{\max} \to \infty$ or even is $\theta_{\max} = \Omega(T)$). We discussed this in detail in Rem1 after Thm5, as well as empirically validated in Sec5.

We hope that was helpful. Please let us know if you need further explanations.

Thanks for your attention. Authors