Bilevel Reinforcement Learning for Stock Data with A Conservative TD Ensemble

1. Lack of Discussion on Non-Stationarity.

(1) "As I understand it, the paper attempts to mitigate non-stationarity through specific data transformations."

We would like to clarify that our approach mitigates the non-stationarity issue primarily from both the algorithmic and data perspectives, rather than relying solely on specific data transformations. The fundamental idea behind the bilevel optimization algorithm is to prevent overfitting during offline learning, ensuring that the model does not simply memorize the optimal policy based on specific patterns in the training set. Different data transformation methods are treated as plug-and-play modules within the bilevel learning framework, designed to generate meaningful augmented data that helps evaluate the robustness of the learned policies.

Notably, the proposed bilevel optimization scheme was shown effective even without any specific data transformation methods (see Table 3). Below, we summarize the ablation study results:

(2) "The rationale behind the choice of these three particular transformations is unclear."

The data transformation methods employed in our work are designed to simulate out-of-distribution yet plausible market changes that have not been included in the training set. From a time series analysis perspective, $F_{1,2,3}$ covers the short-term randomness, long-term trends, multi-scale seasonal patterns of a dynamic system, respectively. Please refer to our General Response 1 for further details.

(3) "Applying this approach in the out-of-sample leads to the future information issue."

We would like to clarify that there is no test data leakage in any of the data transformation methods used. For further details, please refer to our General Response 1.

2. Input Construction and Covariance Matrix

$o_{t-H+1:t}^\text{cov} \in \mathbb{R}^{|S|\times|S|}$ calculates the covariance matrix of returns between pairs of stocks over a fixed period of time $H$ before $t$. In practice, we adopted the settings used in StockFormer. For daily-level stock trading data, we set $H=1$ to capture short-term predictive features and cross-stock relational features, and $H=5$ to capture long-term predictive features. This matrix can partially capture the correlations between stocks and remains operational on larger datasets.

Additionally, we have conducted experiments on another dataset containing more stocks (587 in total). The results indicate that our method is also scalable to larger datasets. Please see our General Response 3 for details.

We would like to thank the reviewer for the prompt reply and ongoing discussion!

1. Further explanation on transformation method $F_1$.

First, following previous RL-based stock trading methods, our trading operations only include buying, holding, or selling, without considering short-selling. Thank you for your valuable suggestion regarding the action space allowing for both long and short positions. We believe this is a significant point, and it will be discussed in future work.

In our work, the data transformation method $F_1$ is primarily designed to prevent overfitting to the in-domain optimal but actually impractical policies learned exclusively on training data. As shown in Table 3 in the manuscript, compared to "$*w/o*$ Data Transformation", the sharpe ratio (SR) improves significantly with the $F_1$ (1.44 $\rightarrow$ 1.53). Additionally, we further test the maximum drawdown (MDD), which decreases by $16 \\%$ (0.25 $\rightarrow$ 0.21), indicating that this approach can mitigate the trading risk.

We were unsure whether inverting the bottom $10\\%$ would yields the same improvement in policy robustness, as short-selling was not included in the action space. But still, following the reviewer's suggestion, we add an experiment that inverts both the top and bottom $10\\%$ of stock returns. We present the comparison results under the online adaptation setup on the CSI-300 dataset below. The results can be fairly compared with those in Table 3 in the manuscript.

2. Clarification on covariance matrix.

We sincerely apologize for the previous explanation regarding the covariance matrix, which may have caused the misunderstanding that only a short historical period was used for its calculation. In our previous response, the fixed period $H$ refers to the future time interval used to calculate stock price changes, not the number of lookback days.

In fact, we use the "price change matrix" (i.e., the daily price changes of all stocks over 252 historical trading days) to compute the covariance matrix. Below, we present the formulas used in the calculation process.

Predefined parameters:

• Lookback length $T$ (252 days in practice),
• Fixed period length $H$,
  • $H=1$ to capture short-term predictive features and cross-stock relational features,
  • $H=5$ to capture long-term predictive features
• Number of stocks $|S|$.

The price change matrix $R_{tj}\in \mathbb{R}^{T\times|S|}$, for $r_{tj}\in R_{tj}$,

$$r_{tj} = \frac{P_{t,j} - P_{t-H,j}}{P_{t-H,j}}$$

The covariance matrix $\Sigma\in\mathbb{R}^{|S|\times|S|}$, for $\Sigma_{ij}\in \Sigma$

$$\Sigma_{ij} = \frac{1}{T-1} \sum_{t=1}^{T} \left( R\_{ti} - \bar{R}\_i \right) \left( R\_{tj} - \bar{R}\_j \right)$$

On the other hand, as you point out, the lookback length is likely to impact the calculation of the covariance matrix, which in turn can affect the final results. To further analyze the impact of lookback length, we conduct additional experiments and present the offline evaluation results on the CSI-300 dataset as follows:

As we can see from the above results, an insufficient lookback length significantly affects the estimation of the covariance matrix, ultimately leading to a noticeable decline in performance. Additionally, the lookback length we selected allows for a more accurate estimation of the covariance matrix, thereby ensuring the stability of the model’s performance.

3. Regarding minor suggestions, comparing the proposed algorithm with ensemble Q-learning methods may enhance the quality of this paper.

Thank you for your suggestions. We conduct new experiments by replacing the ensemble method in our model with the methods [1,2] compared in General Response 2, i.e., using the minimum and mean value of five parallel Q-networks as the Bellman target. The results are presented below, and are included in Appendix G in the revision.

1. Related work on offline RL and offline-to-online RL

Thank you for your encouraging comments and valuable suggestions! We have included a further discussion on the related work in Section 5 in the revision.

2. Description about the proposed method in Introduction.

Thanks again for your suggestion! We have updated the descriptions of "in-domain profits" and "outer-loop optimization" in the revised Introduction section.

3. How does the proposed method (and other baseline) determine the trajectory length?

For the baseline models, we directly used their official implementations and adopted the trajectory length settings as specified in their original literature. For our approach, we followed the same trajectory length as used in StockFormer.

Specifically, during the training phase, the trajectory has two termination conditions: reaching a maximum predefined length or the last day of the training set. In the original manuscript, we set the same configuration as StockFormer, defining a maximum length threshold of 1000 with a random start time. Additionally, we conduct experiments to analyze the impact of using different trajectory lengths threshold in our model (corresponding to the experimental setup in Table 1).

1. Relying on training the policy with real and augmented data, followed by fine-tuning on real data to address the OOD problem in the stock market, is not an ideal approach.

(1) Do the F2 and F3 methods mentioned in the paper potentially introduce data leakage? Is the choice of the top 10% in F1 (why specifically 10%?) reasonable?

Please refer to our General Response 1.

(2) The effectiveness of finetuning of RL-for-finance models.

Thank you for the insightful comment! We completely agree with the reviewer that in practical RL-for-finance tasks, the naive fine-tuning approach often fails to enhance model performance on test data. This is primarily due to overfitting to specific data patterns when fine-tuning on more recent data. This is precisely why we propose the bilevel optimization approach for the RL method!

Theoretically, the bilevel optimization scheme can significantly enhance the model's generalizability to new data. Similar approaches, known as model-agnostic meta-learning (MAML) [1], have been widely adopted to improve finetuning results in few-shot learning scenarios. Intuitvely, it aims to find optimal parameters initialization that can be quickly adapted to a new related task using only a few data and a few gradient steps.

(3) Finetuning vs. Train-from-scratch.

We compare the performance of different RL methods with and without finetuning, using the same configuration as Table 1. Below, we present the Cumulative Return (i.e., PR in the original manuscript) on the CSI-300 dataset. The results are averaged over three random training seeds. Notably, our bilevel optimization approach significantly improves the finetuning results (by +13.39%), while the previous RL approaches do not support such effective model finetuning (e.g., by +0.81% for StockFormer). We include full comparisons in the revised appendix.

Reference: [1] Finn C., Abbeel P., Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. ICML 2017.

2. On the experimental setup.

(1) Data validity: The experimental data only goes up to 2022, without including more recent data (e.g., up to 2024).

In our original experiments, we used data up to 2022 to ensure a fair comparison with StockFormer, which follows the same training and testing period division.

As suggested by the reviewer, we conduct additional experiments using data beyond 2022. In this experiment, we do not extend the training set range but directly test on the CSI-300 dataset spanning from 2022-05-01 to 2024-05-01. During this period, the overall market is weaker than that in the original test set before 2022. Consequently, the annualized returns of all methods are reduced. Nonetheless, our method consistently outperforms all baselines, highlighting its potential for profitability even under more challenging market conditions. We have included these results in Appendix F.3 in the revision.

(2) Stock selection and market representation.

Please refer to our General Response 3.

(3) Evaluation metrics.

Thank you for your suggestion! We have included the following MDD results in the revised Appendix F.2. We summarize the results below (lower is better):

1. On clarity of the notations.

(1) Details about technical indicators

The technical indicators mentioned in the paper follow the settings used in StockFormer. Specifically, we utilized the Stockstats package, and the technical indicators are presented below. We include this in the Appendix E in the revised paper.

(2) State space on Page 2

Thank you for pointing this out. The three types of latent states should be represented as $h_t^\text{relat}$, $h_t^\text{long}$, $h_t^\text{short}$. We have corrected this in the revision.

(3) Dimension of $h_t$ on Page 3

$D$ represents the dimension of the output by the feature extraction module.

(4) Daily prices in Appendix A

As suggested by the reviewer, we have corrected the notation for "closing price" to $o_t^{close}$ in the revision to align with the main text.

(5) Metrics in Appendix C

Thank you for your suggetion. In the revision, we have updated equations (7), (8), and (9) in Appendix C. We have provided a more detailed explanation of these notations and clarified the calculation of each metric.

2. Rationale for the chosen transformations.

Please refer to our General Response 1.

3. Novelty of ensemble-based conservative TD target.

Please refer to our General Response 2.

4. To provide a clearer picture of the proposed algorithm’s stability.

(1) Standard deviations

As suggested by the reviewer, we have added the standard deviations for the experiments reported in Tables 3, 4, and 5.

(2) Rolling training/test set

In Lines 360--368 in the original manuscript, we have evaluated MetaTrader under the online adaptation setup, where all compared models are continuously finetuned on the streaming test data. Implemenatation details are provided in Appendix B. We understand this as using a rolling approach for model updates and testing. If we have misunderstood the reviewer's meaning of "rolling training/test," we kindly request more specific experimental suggestions.

(3) Maximum drawdown

To further showcase the stability of our approach, we have included the MDD (Maximum Drawdown) metric in Table 11 in the revised appendix, as an indicator of downside risk over a specified time period. We summarize the results below:

2. To Reviewer MUZC and jZAh: The differences between our conservative TD method and other ensemble-based Q-learning methods

While the mathematical expressions of our method and other existing methods appear similar, significant differences exist in the network model and data input. Notably, existing ensemble-based Q-learning methods [1-3] typically utilize multiple target Q-networks (with separate model parameters) and compute ensemble value regularization by exploiting the implicit diversity among these Q-networks. In contrast, our approach relies on a single target Q-network and derives the worst-case Q-value by leveraging the explicit diversity introduced through transformed data.

Specifically:

• Existing methods are based on model diversity: Most existing ensemble-based methods require multiple Q-networks with identical input data $(s_{t+1}, a_{t+1})$ to calculate a conservative TD target, as shown in Eq. (1) below, which significantly increases the model size. For example,
  • [1] uses the minimum value of multiple parallel Q-networks as the Bellman target;
  • [2] stabilizes Q-learning by averaging previously learned Q-values as the target;
  • [3] averages all Q-values, excluding those with the highest $N-K$ values.
• Our method is based on data diversity: Our ensemble method is based on original stock data and its transformations $(s_{t+1}^{(n)}, a_{t+1}^{(n)})$ to calculate a conservative TD target by a single Q-function, as illustrated in Eq. (2) below. This approach leverages transformations of stock data to account for diverse market conditions, thereby capturing more variability in the decision-making process. We illustrate the feasibility of our approach from the perspective of Bellman Equations within our offline RL setup (Eq. 4 in the manuscript).

$\hat{Q}^\prime (s\_t, a\_t) = r\_t +\gamma\big[ -\lambda \log \pi\_{\theta} ({\hat{a}}\_{t+1} \mid {s}\_{t+1}) + \underset {k=1,\dots, M}{\Psi} {Q\_{\bar{\phi}\_k}({s}\_{t+1}, {\hat{a}}\_{t+1})}\big]$ (1)

$\hat{Q}^\prime (s_t, a_t) = r_t +\gamma\big[ -\lambda \log \pi_{\theta} ({\hat{a}}\_{t+1} \mid {s}\_{t+1}) + \underset {k=1,2}{\min} \ \underset {n=1:N}{\min} \big({Q\_{\bar{\phi}_k}({s}\_{t+1}, {\hat{a}}\_{t+1})}, Q\_{\bar{\phi}\_k}({s}\_{t+1}^{(n)}, {\hat{a}}\_{t+1}^{(n)})\big) \big]$ (2)

References:

[1] Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble, NeurIPS 2021.

[2] Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble, CoRL 2021.

[3] Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance, NeurIPS 2022.

3. To Reviewer hygz, jZAh: Experiments on a larger market scale

(1) Why is using a larger market scale challenging?

In this work, we focus on advancing RL-based stock trading. To the best of our knowledge, existing RL-based stock trading methods, such as FinRL, StockFormer, and SARL, primarily conduct experiments on relatively small-scale datasets. We attribute this limitation to two main factors:

• Data perspective: Trading suspensions frequently occur in real-world stock data. Previous studies often select stocks based on the requirement that the proportion of valid data exceeds a specific threshold (e.g., 98% in StockFormer) to reduce noise from excessive data interpolation.
• Algorithm perspective: As the stock pool size increases, the action space grows significantly, making it more challenging for RL methods to manage. If we aim to trade thousands of stocks in the market, the dimensionality of the action space can be even larger than the number of training sequences! The difficulty of high-dimensional action space is well-documented in other domains beyond stock trading [4,5]. However, learning an RL-for-stock model with a filtered small stock pool remains valuable. In practice, we can often manually pre-select a set of promising stocks using prior knowledge, enabling more focused and effective RL training.

(2) Additional results on an expanded dataset with 587 stocks

Nevertheless, we conduct experiments on a larger dataset by expanding the range of CSI stocks and selecting a dataset containing 587 stocks. We maintain the same experimental setup as in the offline evaluation described in the manuscript and compare our method with several baselines. The results are presented in the table below.

References:

[4] Action Branching Architectures for Deep Reinforcement Learning, AAAI 2018.

[5] POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition, ICML 2023.