Thanks for you response.

• Please see line 192.

Firstly, you claim that you define a distribution to generate a^_t, but I don't see a^t in your Eq. 5 but an a\tau. This is the reason why I comment "Confusing notation that the decision function f generates a or a*". Secondly, in your response you claim that you explicitly generate a^* with Eq. 3, I saw that as well. This is also why I'm confused about you claim in line 192. And I'm still confused if you want to keep line 192 unchanged.

• There is no f in Eq. 7, do you mean Eq. 5?

• Thanks for your clarification, if such intuitive explanation is not included in the paper, the authors should put it into the paper somewhere obvious for clarification.

• "However, since these trajectories are simulated and the environment is usually known, the optimal action $a^*$ can be determined using Equation 3." First, it's a strong assumption that the environment is known. Do you study the specific case that the environment is known? In addition, I don't see the causality of this sentence. Can you explain more why the trajectories are simulated and the environment is "usually" known, then a^* can be determined?

To me defining a^* as maximizing the reward is fine. But I don't think it's anything to do with simulated trajectories or known environments.

Weakness 2: In the paragraph, below Eq. 9, the authors claim that OOD in this setting is because...

We hope that after addressing Weakness 1, the reviewer will understand why the difference in the trajectory between the pre-training stage and the evaluation stage constitutes an OOD issue. Indeed, such an OOD issue has also been mentioned in [1] (the paper reviewer cited, page 5 right before section 4 in the arxiv version). For clarity, let us explain further where the OOD issue come from.

In the pre-training phase, the trajectories are generated by some behavior function $f$, and the trajectory $(X_1, a_1, o_1,\cdots, X_{t-1}, a_{t-1}, o_{t-1}, X_t)$ (notice that $a_{\tau}$ for $\tau = 1,\cdots, t-1$ is generated by $f$) is accompanied by an optimal decision $a_t^*$. The intention is to let the Transformer learn the best action when it sees the trajectory. However, during the evaluation phase, the trajectories are generated incrementally by the trained transformer $TF_{\theta}$. It is highly likely that this type of trajectories encountered during evaluation have not been seen during pre-training, because that requires the transformer $TF_{\theta}$ to be very close to the behavior function $f$ . Consequently, the Transformer $TF_{\theta}$ might not know the right action when the input is a trajectory it generated itself.

We believe this is clearly an OOD problem. As the reviewer mentioned, it can be thought of as a change in environmental parameters or a shift in tasks between training and testing, which will change the distribution of input trajectories.

Above Eq. 9, pertraining -> pretraining, and this paragraph is unclearly written. Overly referring to the equation make me hard to follow the idea and intuition.

Thanks for the suggestion, we will update it in the next version.

Question 1: The theory is comprehensively covered, while simply adding a mixed training phase and testing

We hope that after addressing Weakness 1, the reviewer will understand why our work is different from other works. In general, we are the first to propose the mix-pretraining scheme, which works empirically and is supported by theoretical results regarding convergence and regret.

First, we disagree with the reviewer's comment that "the theory is comprehensively covered." In [1], the authors mention the OOD issue but do not address it. The plain scheme in [1] does not converge in our setting, where there is a massive number of environments, as shown in Figure 2(a). Moreover, we provide a more meaningful regret analysis with less restrictive assumptions, which could be important for the theory RL/decision-making community.

Next, [2] and [3] focus more on the empirical side. The authors do not mention the OOD problem for the input trajectory, and there are no theoretical results provided. We believe that the reviewer should not downplay the importance of theoretical contributions.

Question 2: In addition, given that Transformer is a powerful sequence model, isn't studying pretraining on state-free bandit problems less meaningful

We believe this is another misunderstanding of our setting. Bandit problems are state-free, but in terms of learning, the information in the current trajectory contains meaningful data (e.g., how many times each arm has been pulled and the historical rewards). The trajectories heavily depend on the behavior function $f$ that generates the pre-training trajectory. Therefore, it is definitely related to sequential problems, as the trajectory containing historical information changes as the transformer keep making online decisions over time. Other decision-making problems, including pricing and the newsvendor problem, follow the same reasoning.

Conclusion

To summarize, we believe there is a significant misunderstanding of our paper. We hope our explanation helps clear up these confusions and would appreciate it if the reviewer could take more time for re-evaluation.

We thank the reviewer for taking the time, but we strongly disagree with these comments. It seems there are fundamental misunderstandings regarding the initial setup of our paper, and we are surprised to see such misunderstandings arise in this well-known RL/decision-making setting. We hope our explanation helps clear up these confusions and would appreciate it if the reviewer could take more time for re-evaluation.

We would like to first address the most fundamental setup question, Weakness 1, as it significantly affects the understanding of the rest of the paper.

Reviewer’s Weakness 1:

The notation is confusing. For example, the decision function $f$ generates $a$ or $a^*$. In Eq. 5, the authors claim to generate $a^*$, but there is no $a^*$ in Eq. 5. I can only infer that $f$ generates $a^*$. Then, below Eq. 7, it becomes $a_\tau$.

Our Reply on Weakness 1:

We are not abusing the notation; it is both precise and correct.

We NEVER claim that $f$ generates $a^*$ in Equation 5. In fact, we explicitly stated TWICE that the optimal decision $a^*$ is generated by Equation 3 (see lines 178 and 195). Additionally, we clearly specified that $f$ is a pre-specified decision function that generates $a_t$ in trajectories, which is commonly known as the behavior function. Could the reviewer please point out where we supposedly state that $f$ generates the optimal decision $a^*$?

Then, in Eq. 7, $f$ generates $a_\tau$ for $\tau = 1, \cdots, t-1$, because we need a trajectory for every $t = 1, \cdots, T$, and these indices are necessary for each $t$. That is why we use $\tau$ in line 195. We believe this is the most appropriate definition and do not see any part that might cause confusion.

To clarify the general intuition further, $f$ can be thought of as a generating function that produces trajectories for pre-training. The actions in these trajectories are often suboptimal. However, since these trajectories are simulated and the environment is usually known, the optimal action $a^*$ can be determined using Equation 3.

Thank you for your quick reply. Please see the response below and let us know if there are further confusions.

• Question: For line 192. Firstly, you claim that you define a distribution to generate $a_t^*$, but I don't see $a_t^*$ in your Eq. 5.

Response

192 is a long sentence, and we claim that we define a distribution $P_{\gamma, f}$ that generates BOTH $H_t$ and $a_t^*$, where the full specification of $H_t$ (with the necessary notation $a_{\tau}$) is defined in line 193-194, which is Eq. 5; and $a_t^*$ is followed immediately in line 195. We think it follows the natural order for introducing the setups.

• Question: For line 192. Secondly, in your response you claim that you explicitly generate $a_t^*$ with Eq. 3, I saw that as well. This is also why I'm confused about you claim in line 192. And I'm still confused if you want to keep line 192 unchanged.

Response

I think the confusion comes from the fact we are introducing $H_t$ (which involves $a_{\tau}$ for $\tau = 1, \cdots. t-1$) and $a_t^*$ together. We can update the new version where $H_t$ and $a_{\tau}$ are introduced separately. We hope this would make things more clear. Thank you for the suggestion, and please let us know if you have some better ways to do so.

• Question: There is no $f$ in Eq. 7, do you mean Eq. 5?

Response

Yes we mean Eq.5. Sorry for the confusion.

• Question: Thanks for your clarification, if such intuitive explanation is not included in the paper, the authors should put it into the paper somewhere obvious for clarification.

Response

Thank you for the suggestion, we will be adding another graph with explanations in the updated version.

• Question: "However, since these trajectories are simulated and the environment is usually known, the optimal action can be determined using Equation 3." First, it's a strong assumption that the environment is known. Do you study the specific case that the environment is known? In addition, I don't see the causality of this sentence. Can you explain more why the trajectories are simulated and the environment is "usually" known, then a^* can be determined?

Response

Thank you for raising this question. We are glad to see that we are discussing the core theme of this paper. Generally speaking, our approach represents a different design philosophy for learning decision-making algorithms, one that is more data/simulator-driven and relies on the learning power of Transformers on large corpora of data.

In the traditional UCB setting, we assume that the environment follows a specific class of distributions (e.g., Gaussian). Based on this assumption, UCB bounds are developed, leading to good theoretical regret and empirical performance. However, this also means that for different distributions and structures (e.g., stochastic bandit environments vs. linear bandit environments), different algorithms are required (e.g., UCB vs. LinUCB [1]). Therefore, if at test time we do not know which environment the data is sampled from, there is no established consensus on which algorithm to apply.

In our design philosophy (illustrated in Figure 1), we sample a large number of environments and sample trajectories from them (denoted as the simulator $\mathcal{S}$ in Figure 1). Although at test time, the current environment $\mathcal{E}$ may be unknown at the initial stage, as time progresses and more trajectory data is observed, the Transformer can identify which pre-training environment the current trajectory most closely resembles. Specifically:

• If the current environment is part of $\mathcal{S}$, the Transformer leverages pretraining data from that environment to generate optimal decisions.
• If the current environment is not part of $\mathcal{S}$, the Transformer generalizes to make an "educated guess" for the optimal decisions.

Thus, we adopt a simulator-based approach with a large number of environments, which is why the optimal actions $a_t^*$ are known during pre-training. The rationale is that the Transformer can effectively learn to generalize and adapt under a wide variety of trajectories, leading to strong empirical performance.

In Figure 12, we demonstrate the performance of the Transformer, pretrained on two different types of environment distributions with different structures (analogous to stochastic bandit vs. linear bandit) in a decision-making setting. These results highlight the effectiveness of our approach in scenarios where traditional methods are suboptimal.

[1] Li, Lihong, et al. "A contextual-bandit approach to personalized news article recommendation." Proceedings of the 19th international conference on World wide web. 2010.

Thank you for raising these suggestions and replying quickly. We truly appreciate your time. We still have two questions regarding your reply.

First Question

First, we would like to mention that there might be some misunderstanding regarding the claim of our contribution to decision-making. Sequential decision-making problems are not equivalent to RL and include many other problems with practical impacts, such as pricing, inventory management, and control of service systems. We understand that the reviewer might come from a different area, but if you are interested, we kindly invite you to take a few minutes to search these papers [1][2][3], which demonstrate that these settings are long-standing and common in decision-making with practical significance.

In short, these problems differ from RL in that state transitions are sometimes deterministic. For example, after making a decision on inventory replenishment, the state on the next day will deterministically depend on the decision made the previous day, and the randomness comes from other areas, for example the demand in a day. While this represents a simpler version of RL, it still involves transition dynamics and is more complex than stochastic bandit problems due to constraints (such as inventory levels) that affect the decision space.

In our paper, we conduct extensive experiments on pricing and newsvendor problems, which we believe are more complex than simple bandit problems. Given these clarifications, we hope this addresses the reviewer’s concerns about the perceived "over-claim."

Second Question

We thank the reviewer for the suggestion. We are curious to know: if we address the reviewer’s feedback during the rebuttal period, would you consider raising the score one more time? There are certain issues we believe could be easily fixed in the main text in terms of writing. Additionally, for further discussions or additional numerical studies, we could provide updates in a separate file and incorporate final changes into the main text if everything is resolved.

Conclusion

We thank the reviewer again for the time and effort in this round of review. We look forward to receiving further feedback.

[1] Petruzzi, Nicholas C., and Maqbool Dada. "Pricing and the newsvendor problem: A review with extensions." Operations research 47.2 (1999): 183-194

[2] Den Boer, Arnoud V. "Dynamic pricing and learning: historical origins, current research, and new directions." Surveys in operations research and management science 20.1 (2015): 1-18.

[3] Besbes, Omar, and Assaf Zeevi. "Dynamic pricing without knowing the demand function: Risk bounds and near-optimal algorithms." Operations research 57.6 (2009): 1407-1420.

According to the reviewers request, we conduct another round of experiment, hoping to provide some insights for the reviewers question.

The reviewer would like to see "How the distribution mismatch between the pre-training". We conducted another set of systematic experiments on this topic.

Problem setting

We take the dynamic pricing problem to illustrate this, given that i) the exploration-exploitation of the dynamic pricing problems is a bit more complex compared to bandit problems; and ii) there are more baseline algorithms to compare with than those of the bandit algorithms.

We define the demand function in the pricing problem as $D(a) = \alpha^\top X - \beta^\top X \cdot a + \epsilon,$ where $(\alpha, \beta)$ represents the parameter set associated with the environment, $X$ denotes the contextual features, $\epsilon$ is random noise, and $a$ is the continuous decision variable corresponding to price. The reward is given by $a \cdot D(a)$. Over a horizon of $T$, our goal is to dynamically set $a_t$ for $t \in \{1, 2, \dots, T\}$ to minimize regret, defined as $E\left[\sum_{t=1}^T a_t^* \cdot D(a_t^*) - \sum_{t=1}^T a_t \cdot D(a_t)\right],$ where $a_t^*$ is the optimal price in hindsight. For this example, the feature is of dimension $6$.

Source of distributional mismatch

To show the performance under pre-training and evaluation mismatch, We conduct experiments that analyze distribution mismatch in two aspects.

• Adjust the generation of testing environments by altering the distributions of the noise $\epsilon$
• Adjust the (generation of) parameters $(\alpha, \beta)$ to deviate from those in the pre-training environments.

Benchmark algorithms

We compare our algorithm with common off-the-shelf pricing algorithms:

• ILSE: Iterative least square estimation [1]
• CILS: Constrained iterated least squares [2]
• TS: Thompson sampling for dynamic pricing [3]

Experiment I setting

In experiment I, we train the decision-maker under environment with pre-training noise $\epsilon_t\sim\mathcal{N}(0,0.2)$, and vary testing noise distributions such that $\sigma^2 = 0.1$, $\sigma^2 = 0.2$, and $\sigma^2 = 0.3$, separately.

Please see part II for results, details and insights.

Experiment II setting

We further evaluate our model under distributional shifts in the generation of parameters $(\alpha, \beta)$, considering both out-of-domain and in-domain shifts. During pre-training, these parameters are sampled per environment as $\alpha \sim \text{Unif}([0.5, 1.5]^6)$ and $\beta \sim \text{Unif}([0.05, 1.05]^6)$. In the testing phase, two types of shifts are introduced, parameterized by the shift level $\mu_{\text{shift}}$:

• Out-of-domain shifts: Here, the test parameters $(\alpha, \beta)$ can be sampled beyond the original training ranges to simulate scenarios where prior knowledge fails to encompass the true environment space. Specifically, we generate $\alpha \sim \text{Unif}([0.5+\mu_{\text{shift}}, 1.5+\mu_{\text{shift}}]^6)$ and $\beta \sim \text{Unif}([0.05+\mu_{\text{shift}}, 1.05+\mu_{\text{shift}}]^6)$. Four shift levels are evaluated: $\mu_{\text{shift}}=0$ (matching the pre-training distribution) and $\mu_{\text{shift}}=0.1, 0.5, 1$.
• In-domain shifts: For these shifts, sub-intervals of length $(1 - \mu_{\text{shift}})$ within the original training ranges are randomly selected, and parameters are sampled uniformly within these sub-intervals. Specifically, for a given $\mu_{\text{shift}}$, we sample $\kappa_1 \sim \text{Unif}([0.5, 1.5 - (1 - \mu_{\text{shift}})])$ and $\kappa_2 \sim \text{Unif}([0.05, 1.05 - (1 - \mu_{\text{shift}})])$. Then, we draw $\alpha \sim \text{Unif}([\kappa_1, \kappa_1 + 1 - \mu_{\text{shift}}]^6)$ and $\beta \sim \text{Unif}([\kappa_2, \kappa_2 + 1 - \mu_{\text{shift}}]^6)$. This approach simulates a scenario in which the prior knowledge is conservative, assuming a broader feasible space than the true space that generates the testing environment. A larger $\mu_{\text{shift}}$ corresponds to more conservative prior knowledge, implying a larger expected feasible parameter space than the actual one. We consider four shift levels: $\mu_{\text{shift}}=0$ (matching the pre-training distribution) and $\mu_{\text{shift}}=0.1, 0.2, 0.3$.

Please see part II for results, details and insights.

[1] Keskin, N. Bora, and Assaf Zeevi. "Dynamic pricing with an unknown demand model: Asymptotically optimal semi-myopic policies." Operations research 62.5 (2014): 1142-1167.

[2] Qiang, Sheng, and Mohsen Bayati. "Dynamic pricing with demand covariates." arXiv preprint arXiv:1604.07463 (2016).

[3] Wang, Hanzhao, Kalyan Talluri, and Xiaocheng Li. "On dynamic pricing with covariates." arXiv preprint arXiv:2112.13254 (2021).

Hi reviewer hvZS,

We have spent several days conducting an additional set of experiments, and you can find the results here:
https://openreview.net/forum?id=CiiLchbRe3&noteId=40PEGFrve4

We hope this provides a relatively detailed analysis regarding the distribution mismatch between the pre-training and evaluation stages.

We are happy to update the writing as you suggested and include these additional experimental results in the revised version of the paper.

Since the conversation window is closing in 3 days, please let us know if there is anything else you would like to know. We also hope the reviewer can re-evaluate our work based on these updated results.

Weakness 3: Additionally, while the insights generated are interesting, the paper lacks a clear and compelling rationale for why the pre-training is worth the extra computation compared to simpler methods...

In terms of this weakness, we think the reviewer may have overlooked two important factors.

1. General Algorithm For Decision-Making

This algorithm is not specifically designed for bandit problems. It is a pipeline intended to be broadly applicable across various settings, including stochastic bandits, linear bandits, dynamic pricing, and newsvendor problems. The general pipeline has its merit in designing unified frameworks as a step toward artificial general intelligence (AGI).

Moreover, the theoretical analysis is a fundamental step in understanding Transformers for decision-making problems. If we do not analyze it within the bandit setting as a foundation, how can we extend it to more general RL or practical applications? For instance, the well-known online RL algorithm UCBVI [1] builds on the development of UCB in bandit problems. Therefore, we believe that studying the transformer framework in this simpler setting is both meaningful and necessary.

2. Applicability to Complex Decision-Making Problems

We would like to highlight that our pipeline is capable of handling more complex yet practical problems beyond simple bandits. In real-world decision-making scenarios, such as dynamic pricing and newsvendor problems, it is often necessary to switch between different models depending on the environment. To make an analogy to the bandit setting, an environment may alternate between a linear bandit problem (requiring the LinUCB [2] algorithm to perform well) and a stochastic bandit problem (requiring the UCB algorithm). A unified framework like our transformer-based approach effectively handles these environments by identifying the environment and selecting the right algorithm when seeing the current trajectory, as demonstrated in Figure 12. This versatility extends the applicability of our method beyond simple bandit problems to more complex decision-making tasks.

Weakness 4: Overall, math notation is confusing and seems even incorrect/inconsistent.

We would appreciate it if you could provide specific examples of the issues you found. Raising such concerns without explicitly identifying the problems makes it challenging to address them and, in our view, is both irresponsible and unprofessional. Clear feedback would help us resolve any potential misunderstandings more effectively.

Question 1: Do the benchmarks used need pre-training? If not, then what would the authors argue...

This represents a different design philosophy. Previously, for every problem instance, problem-specific algorithms (which are the benchmark algorithms) had to be developed, such as UCB for bandits and LinUCB [2] for linear bandits. We propose a unified framework that is pretrained on trajectories sampled from a massive number of environments. In the evaluation phase, by observing the current trajectory, it effectively identifies the environment and selects the "right" algorithm.

This unified approach is successful across various decision-making problems, as demonstrated in our numerical examples. It is more flexible in terms of algorithm design, provides better performance than problem-specific benchmarks (figure 4), and is generalizable as the number of environments in the simulators $\mathcal{S}$ increases, allowing us to handle more complex environments (figure 12).

Question 2: In line 167, it is stated that...

We would like to point out that the pretrained transformer is trained on a massive number of trajectories from a large number of environments, whereas previous methods that incorporate prior knowledge from previously collected data—such as replay buffers, offline learning, and transfer learning—cannot effectively leverage this heterogeneous offline data without knowing the current environment. It is straightforward to construct counterexamples where trajectories from unrelated environments adversely impact algorithm performance.

Developing algorithms that can effectively leverage this type of offline data in a online manner requires a dedicated study, which current offline RL and transfer learning techniques do not adequately address. In this setting, our unified framework, pretrained on trajectories sampled from a wide range of environments, can effectively identify the environment and select the "right" action.

References

[1] Azar, Mohammad Gheshlaghi, Ian Osband, and Rémi Munos. "Minimax regret bounds for reinforcement learning." International conference on machine learning. PMLR, 2017.

[2] Li, Lihong, et al. "A contextual-bandit approach to personalized news article recommendation." Proceedings of the 19th international conference on World wide web. 2010.

We appreciate the reviewer’s questions and suggestions. It seems there may be some misunderstandings about the specific setup and other details of our paper. We hope our response clarifies these points and kindly invite the reviewer to re-evaluate our contributions in light of this clarification.

Weakness 1: The main weakness can be the narrow scope of impact as the sequential decision making problem in the paper is very narrowly defined...

We acknowledge that our setting does not cover the general RL framework. However, we would like to emphasize that this is a unified approach, encompassing nearly all other settings—such as various types of bandits, dynamic pricing, and inventory control. These combined research areas already represent a significant and impactful community.

Moreover, this work represents an essential step toward artificial general intelligence (AGI), which holds significant implications for the future of AI. If we do not analyze it within the bandit setting as a foundation, how can we extend it to more general RL or practical applications? For instance, the well-known online RL algorithm UCBVI [1] builds on the development of UCB in bandit problems. Therefore, we believe that studying the transformer framework in the bandit setting is both meaningful and necessary.

Weakness 2a: The text does very little to motivate the need for the Learned Decision Function (LDF)...

We think there might be some misunderstanding regarding the setup. Generally speaking, the Learned Decision Function (LDF), which is the transformer-based decision-maker, is defined in Section 2, and the pretraining method aims to generate an LDF with good performance. The section 4 analyzing LDF is necessary because we need to analyze the theoretical performance of the transformer-based decision-maker, such as the realizability and the regret bound, which are very common and necessary results in the decision-making literature (e.g., bandits, pricing). We were indeed surprised by this question asking for motivations.

To clarify the relationship: In Section 2 (Page 3, Line 126), we define the LDF, which generates actions from the Transformer as $a_t = TF_{\theta}(H_t)$. The training loss is generally defined as $\mathbb{E}[\sum_{t=1}^T l(TF_{\theta}(H_t), a_t^*)]$ (Page 5, Line 220).

In Section 3, we discuss properties of the pretraining and generalization loss, while in Section 4, we analyze the properties of the actions generated by the LDF. Specifically, we identify the Bayes-optimal decision function (Page 7, Line 367), which is the optimal decision. We then show in Proposition 4.1 that the Bayes-optimal decision function minimizes our training loss. This implies that by minimizing the empirical loss, the trained transformer is likely to perform close to the Bayes-optimal decision function. Based on this, Proposition 4.4 provides a regret analysis for the LDF, which is a core component of almost every theoretical decision-making paper.

Therefore, the analysis of the LDF is fundamental to demonstrating important theoretical properties related to decision-making. We hope this clarifies why the LDF is essential.

Weakness 2b: It seems unlikely that there is not an acceptable baseline that can be used from the literature. The confusion is further amplified in the experiments section, where the authors use different baselines to evaluate the pre-trained methods...

In summary, the only baseline in [36], from which our method is developed, does not converge and therefore cannot be used as a baseline. Additionally, we cannot use the same baseline across different problems such as stochastic bandits, linear bandits, pricing, and the newsvendor problem, because each problem is distinct, requiring problem-specific algorithms as baselines.

For the first point, our method, Pre-trained Transformer (PT), originates from [36], which introduces a new architecture and serves as the only benchmark baseline. A key contribution of our paper is improving the pretraining pipeline from [36], enabling our improved PT to converge when pretrained on a large number of environments, while the method in [36] fails to do so (see Figure 2(a)). Therefore, it does not make sense to use the only available baseline in the literature, as it does not converge.

For the second point, we cannot use the same baseline for stochastic bandits, linear bandits, pricing, and the newsvendor problem because these problems differ significantly, and only problem-specific algorithms can serve as appropriate baselines. For instance, UCB for bandit algorithms cannot be directly applied to pricing settings.

Reference

[1] Azar, Mohammad Gheshlaghi, Ian Osband, and Rémi Munos. "Minimax regret bounds for reinforcement learning." International conference on machine learning. PMLR, 2017.

[36] Lee, Jonathan, et al. "Supervised pretraining can learn in-context reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

Hi reviewer uvAS,

We hope this message finds you well. Since we haven’t heard back from you in a while, we wanted to remind you that the conversation window is closing in 3 days.

Additionally, as the reviewer hvZS suggested, we have spent several days conducting additional experiments, which provide a detailed analysis of the distribution mismatch between the pre-training and evaluation stages.

You can find the results here: https://openreview.net/forum?id=CiiLchbRe3&noteId=40PEGFrve4

We are happy to update the writing as you suggested and include these additional experimental results in the revised version of the paper.

Please let us know if there is anything else you would like to know. We also hope you can re-evaluate our work based on these updated results.

We appreciate the reviewer’s questions and suggestions. It seems there may be some misunderstandings about the specific setup and other details of our paper. We hope our response clarifies these points and kindly invite the reviewer to re-evaluate our contributions in light of this clarification.

Weakness 1: My biggest concern is the lack of discussion ...

We do not see significant issues with respect to the computational and memory overhead. In terms of computational overhead, our model scales linearly with respect to the context length, as is the case with many generative language models, which is totally acceptable. In terms of memory, our transformer adopts the GPT-2 architecture, which requires less than 1 gigabyte of VRAM and can run on almost all GPUs compatible with CUDA. Therefore, we don’t think there is a significant issue in terms of inference speed or memory requirements.

Weakness 2: The comparison between the proposed method and UCB/TS seems unfair, as the transformer benefits from...

We believe there may be a misunderstanding of our setup. Our transformer is pretrained on a vast number of environments, with each environment generating hundreds of thousands of trajectories. In the evaluation or online learning phase, if the current environment and its sampled trajectories were known, we could leverage that data in combination with the UCB/TS algorithms. However, since the environment is unknown beforehand, using the large volume of offline data indiscriminately could potentially degrade performance. While some algorithms may leverage massive pretraining data from heterogeneous environments online, fully understanding their analytical and computational properties would require a dedicated study, which is beyond the scope of this paper. For this reason, we adhere to the original UCB/TS algorithms in our comparisons.

Weakness 3: The claimed short-term advantage of the proposed method also seems unconvincing. While it’s true that...

To address this question, we would like to reiterate our main contribution: a stable and unified pretraining pipeline for transformers in the context of decision-making, with both theoretical guarantees and strong empirical performance. This pipeline is designed to be broadly applicable across various settings, including stochastic bandits, linear bandits, dynamic pricing, and newsvendor problems.

We believe it would be inappropriate to compare our method with problem-specific algorithms, as those algorithms are tailored to individual problem instances rather than offering a unified framework. The strength of our approach lies in its generality and versatility across different problem domains, which is distinct from optimizing performance in specific scenarios.

Weakness 4: I am not very familiar with related work on applying deep learning...

We thank the reviewer for suggesting this, and we will put this in the updated version.

Weakness 5: The clarity and organization of the paper could be improved...

We thank the reviewer for suggesting this, and we will put this in the updated version.

Question 1: What is the difference between the setting described in the paper...

The setting in our paper encompasses both the stochastic bandit and contextual bandit settings. In Appendix B, we provide detailed examples that fall within this framework. We believe this is the most general formulation, as it includes these specific settings. A contextual bandit framework, by contrast, would be considerably less general.

Question 2: What are the disadvantages of using pre-trained transformers compared to structured algorithms like UCB and Thompson Sampling...

The computational cost grows linearly (similar to the UCB and TS algorithms) with respect to the context length, as it corresponds to the number of tokens that need to be evaluated by the large language model, which we believe is entirely acceptable in practice. In terms of memory, our transformer is based on the GPT-2 architecture, requiring less than 1 GB of VRAM and running on most CUDA-compatible GPUs. Therefore, we do not anticipate significant practical issues with inference speed or memory requirements.

Question 3: If UCB/TS had access to the same amount of pretraining data, how would...

As mentioned earlier, for UCB/TS to effectively leverage the pretraining data, they would need to know the current environment and its corresponding trajectories. Without this knowledge, it is straightforward to construct counterexamples where trajectories from unrelated environments could adversely impact UCB/TS performance by distorting the confidence bounds or posterior estimates.

Hi reviewer Svgy,

We hope this message finds you well. Since we haven’t heard back from you in a while, we wanted to remind you that the conversation window is closing in 3 days.

Additionally, as the reviewer hvZS suggested, we have spent several days conducting additional experiments, which provide a detailed analysis of the distribution mismatch between the pre-training and evaluation stages.

You can find the results here: https://openreview.net/forum?id=CiiLchbRe3&noteId=40PEGFrve4

We are happy to update the writing as you suggested and include these additional experimental results in the revised version of the paper.

Please let us know if there is anything else you would like to know. We also hope you can re-evaluate our work based on these updated results.

We thank the reviewer for these comments. Please see our reply below, and we hope this will make things clearer. Please let us know if we have addressed your questions. We would greatly appreciate it if the reviewer could re-evaluate our work based on this round of response.

Weakness 1: I think the setup in Section 2 goes on a bit too long and could be more concise...

We thank the reviewer for the suggestion. We will update it in the next version.

Weakness 2: The framework leans heavily on simulated environments (and) for pre-training, which might limit scalability to real-world cases. Generating high-quality simulations can be both costly and challenging.

We would like to mention that the design philosophy of our approach is quite different from the traditional decision-making approach. We hope the reviewer can read our reply below to gain a better understanding of our approach. In short, our pipeline is different in that:

• It is a unified approach that can solve a wide variety of decision-making problems (same pre-training approach) under different environments (we just sample these environments for pre-training). By sharing a uniform pre-training structure. This does not require users to design instance-specific (bandit vs pricing) or environment-specific (stochastic bandit vs linear bandit) algorithms.
• Although it works in a simulated environment, the transformer architecture is capable of generalizing and extracting specific patterns from the trajectories it has seen. Therefore, as long as the simulator is large enough to include trajectories that is similar to the current environment, we can count on the transformer to quick identify the environment and adapt to the corresponding optimal algorithms.

While we do agree that it is not scalable for complicated RL problems in practice, it is applicable for real-world business problems including dynamical pricing and inventory problems, as suggested in Figure 12.

Details

To be more specific on those two features mentioned above. In the traditional UCB setting, we assume that the environment follows a specific class of distributions (e.g., Gaussian). Based on this assumption, UCB bounds are developed, leading to good theoretical regret and empirical performance. However, this also means that for different distributions and structures (e.g., stochastic bandit environments vs. linear bandit environments), different algorithms are required (e.g., UCB vs. LinUCB [1]). Therefore, if at test time we do not know which environment the data is sampled from, there is no established consensus on which algorithm to apply.

In our design philosophy (illustrated in Figure 1), we sample a large number of environments and sample trajectories from them (denoted as the simulator $\mathcal{S}$ in Figure 1). Although at test time, the current environment $\mathcal{E}$ may be unknown at the initial stage, as time progresses and more trajectory data is observed, the Transformer can identify which pre-training environment the current trajectory most closely resembles. Specifically:

• If the current environment is part of $\mathcal{S}$, the Transformer leverages pretraining data from that environment to generate optimal decisions.
• If the current environment is not part of $\mathcal{S}$, the Transformer generalizes to make an "educated guess" for the optimal decisions.

Thus, we adopt a simulator-based approach with a large number of environments. The rationale is that the Transformer can effectively learn to generalize and adapt under a wide variety of trajectories, leading to strong empirical performance.

[1] Li, Lihong, et al. "A contextual-bandit approach to personalized news article recommendation." Proceedings of the 19th international conference on World wide web. 2010.

Please see the following replies to your questions. We hope our responses make things clearer. Please let us know if we have addressed your questions. We would greatly appreciate it if the reviewer could re-evaluate our work based on this round of responses.

Question 1 part I: How do the authors choose the environments for pre-training, and do variations in the gamma values during training impact the model’s ability to generalize? 

The environments are chosen in the following ways.

• Similar environment with different parameters: This is the most standard setup for pre-training environments. For example, the bandit environments in Figure 4 and Figure 16 are pre-trained using trajectories sampled from one category of distribution (e.g., Gaussian) with different parameter setups (e.g., different means and variances). Our performance under this setting is quite strong.

• Different environment with different structures: We also include pre-training data from different categories of environments. In practice, we may not know if the current environment follows a stochastic bandit environment or a contextual bandit environment. Standard bandit algorithms require prior knowledge of the environment to design specific algorithms for each case. In contrast, our model in the evaluation phase aims to identify the correct class of environment and adapt to the optimal algorithm during evaluation, enabling in-context model selection. The strong empirical results of this approach are shown in Figure 12.

Question 1 part II: It would be interesting to know if a big difference between training and testing gamma values affects performance or creates bias.

If the gamma in the testing environment has been seen in the pre-training environment, our model performs well compared to standard baselines (see Figure 4). If the gamma in the testing environment is very out-of-distribution (OOD), then, as shown in Figure 16, the performance of our approach is slightly affected. However, it is less affected compared to some standard baselines except Thompson Sampling, and still outperforms all of them by a significant margin.

Question 2: Have the authors thought about using curriculum learning, where self-generated actions gradually increase over time, instead of the two-phase switch? I think it would enjoy less theoretical property, but possibly perform better in numerical experiments.

Yes, we used curriculum learning in pre-training. Please see page 36 (Appendix E) for the details in our curriculum training.

Hi reviewer 8XDh,

We hope this message finds you well. Since we haven’t heard back from you in a while, we wanted to remind you that the conversation window is closing in 3 days.

Additionally, as the reviewer hvZS suggested, we have spent several days conducting additional experiments, which provide a detailed analysis of the distribution mismatch between the pre-training and evaluation stages.

You can find the results here:
https://openreview.net/forum?id=CiiLchbRe3&noteId=40PEGFrve4

We are happy to update the writing as you suggested and include these additional experimental results in the revised version of the paper.

Please let us know if there is anything else you would like to know. We also hope you can re-evaluate our work based on these updated results.