OPHR: Mastering Volatility Trading with Multi-Agent Deep Reinforcement Learning

Thank you for your valuable comments. Please find our detailed responses below.

Originality and Related Work Discussion

You are correct in pointing out the work by Tan et al. [1]. We acknowledge this important reference and will revise our claims more precisely. While [1] does apply deep learning to options trading, there are key differences in problem formulation and approach:

Different trading objectives: [1] focuses on option portfolio management, constructing long/short positions across different stocks' straddles to form diversified option portfolios without underlying hedging. Our work specifically targets volatility trading with dynamic hedging for single underlying assets, which involves fundamentally different challenges, including gamma scalping, delta hedging, and path-dependent profit optimization.

Methodological differences: [1] uses supervised learning for portfolio allocation decisions, while we are the first to formulate volatility trading as an MDP problem and apply reinforcement learning to learn optimal volatility timing and hedging strategies jointly.

Market focus: [1] addresses equity options portfolio construction, while we focus on cryptocurrency options volatility trading with continuous hedging requirements.

We will revise our manuscript to more rigorously discuss this related work and clarify that our contribution lies in being the first RL framework designed explicitly for volatility trading with dynamic hedging, rather than general options trading.

Transaction Costs

We do incorporate transaction costs in our simulation:

• Commission fees: Following Deribit's structure - 0.05% for perpetual futures and 0.03% for options (capped at 12.5% of option price)
• Bid-ask spread costs: Implicitly included through realistic market order execution prices in backtesting

Our trading frequency is relatively low (average holding periods of 9-51 hours as shown in Table 3), making transaction costs a small component of total P&L. The following table shows transaction costs as a percentage of total P&L in our experiments:

These results demonstrate that our trading strategy is not sensitive to transaction costs, as the fees represent a relatively small proportion of overall profits. We apologize for not stating this clearly and will add this information to the revised manuscript.

RL Training Stability

We acknowledge the importance of training stability analysis. Our framework incorporates several stability-enhancing design elements:

Oracle initialization: Phase 1 provides consistent starting points, reducing variance across training runs.

n-step TD learning: Effectively reduces noise in Q-value estimation, making RL training convergence more stable than 1-step methods.

Lower-frequency HR-Agent switching: The HR-Agent's less frequent hedger selection (every 24 hours) improves training stability by reducing action space complexity and preventing convergence difficulties from frequent switching.

Relative reward design: Using the advantage of selected hedgers over baseline hedgers as reward provides more stable learning signals than absolute performance measures.

Baseline Performance and Implementation

We provide detailed descriptions of all baseline implementations in Appendix C.3, including consistent principles for position entry/exit, hedging policy, take-profit/stop-loss rules, and holding periods setup.

For each category of baseline strategies, we performed systematic grid search to optimize key hyperparameters as follows:

Single-Factor Models (Mean Reversion / Momentum): Construct percentile-based signals from a 365-day historical RV distribution. Realized volatility is computed over multiple time windows (3–24 hours), and trading is triggered when thresholds on the percentile or its momentum are met.

Machine Learning Models (GBDT, MLP, LSTM): Use predicted annualized RV over a forward window and compare it to current ATM IV to generate trading signals. Entry/exit thresholds were tuned for best validation performance. Fine-tuned take-profit and stop-loss rules were applied.

All baseline strategies were evaluated under the same market environment with performance metrics to ensure fair and consistent comparison.

Despite careful tuning and consistent evaluation, the predominantly negative baseline returns reflect the inherent difficulty of volatility trading rather than implementation issues:

1. Model assumption burden: Traditional methods require explicit volatility modeling choices that introduce misspecification risks, while RL learns optimal actions directly without these constraints.

2. Path-dependent execution gap: Building on these modeling limitations, optimal hedging decisions depend on complex interactions between market conditions, position Greeks, and future price paths (Section 2.2, Figure 1). ML models provide point forecasts but cannot capture the sequential, path-dependent nature where the sequence of hedging decisions critically impacts P&L. Converting forecasts into profitable trading sequences requires manually designed rules that our grid search experiments showed are difficult to optimize across market regimes.

Additional baseline details (DLOT): While [1] trades multiple assets' straddles without delta hedging (using only straddle P&L as labels), we focus on single-asset volatility trading requiring delta hedging. Therefore, we use delta-hedged straddle P&L as prediction labels. This baseline closely resembles our Phase 1 Oracle policy - one uses supervised learning loss, the other uses offline RL loss.

Results analysis: DLOT significantly outperforms RV prediction-based methods, benefiting from an end-to-end design that avoids model assumptions - similar in spirit to our RL approach. However, lacking adaptive hedger selection and further RL optimization, it cannot match our method's performance.

This confirms that even sophisticated prediction-based approaches struggle with the execution challenges our RL framework addresses.

Presentation Improvements

Thank you for these valuable suggestions. We will comprehensively revise the manuscript to address acronym clarity (defining IV, RV, P&L in the main text), format consistency, logical flow reorganization, and comprehensive proofreading to ensure all technical terms are properly introduced. We appreciate your thorough review and will incorporate these improvements to enhance the paper's clarity and accessibility.

Reference:

[1] Deep Learning for Options Trading: An End-To-End Approach. ICAIF 2024.

Thank you for your valuable comments. Please find our detailed responses below.

Innovation from RL Perspective

While our framework builds upon established RL techniques, this work represents the first application of RL to volatility trading in options markets. Our contributions go far beyond naive application of existing methods - we designed a complete framework specifically for this previously unsolved problem:

1. Novel problem formulation: We introduce the first Cooperative MDP formulation for volatility trading (Section 3), decomposing the complex task into specialized agents with complementary objectives that address the unique interdependence between position sizing and dynamic hedging.
2. Tailored multi-agent training framework: The two-phase training with Oracle initialization, alternating agent updates, and relative reward design specifically addresses volatility trading's challenges - delayed rewards, noisy signals, and complex state-action dependencies that don't exist in traditional RL applications.
3. Specialized technical details: Critical implementation details include n-step TD for handling noisy option trading rewards, lower-frequency HR-Agent switching for training stability. These technical choices are essential for making RL work in this domain.

The success of our approach required solving numerous domain-specific challenges from problem formulation through implementation details - simply applying standard RL algorithms would not have succeeded in this complex financial trading environment.

Advanced Volatility Model Baselines

Even sophisticated volatility models like GARCH variants or deep volatility networks face the same fundamental limitations we identified:

1. Model assumption burden: Advanced volatility models still require explicit modeling choices and assumptions that introduce misspecification risks, while RL learns optimal actions directly without these constraints. 2. Forecasting-execution gap: The core issue isn't prediction quality (Table 10 shows respectable performance) but translating forecasts into profitable sequential trading decisions. Advanced models excel at forecasting but cannot capture the path-dependent execution required for options trading.

Empirical evidence: We tested GARCH and DeepVol models:

Despite their sophistication, these models still underperform significantly. The transition from forecasting capability to profitability requires complex model construction and market-specific adjustments - a process our RL method circumvents through direct action optimization without manual model tuning.

Ablation Study on n-step TD Learning

We use n-step TD because 1-step TD struggles with convergence due to noisy stepwise reward signals in options trading. The n-step approach is essential for capturing the delayed and cumulative nature of options trading rewards. In volatility trading, profits don't materialize immediately from individual actions but accumulate over multiple time steps through gamma scalping (buying low and selling high as prices oscillate) and theta decay (time value erosion). A single trading decision's profitability depends on subsequent market movements and hedging actions, creating reward signals that are inherently delayed and path-dependent. In our experiments, 1-step TD fails to adequately learn from these profitable experiences because it cannot connect immediate actions to their eventual cumulative rewards, leading to poor strategy development and convergence difficulties. The n-step TD method effectively addresses this by incorporating future rewards over longer horizons, enabling better learning from the temporal structure of options trading profits.

Theoretical Analysis and Reproducibility

While we don't provide formal convergence guarantees, our framework incorporates principled design elements that ensure performance stability in practice: Oracle initialization, n-step TD learning for noise reduction, lower-frequency agent switching for training stability, and relative reward design for stable learning signals.

Regarding reproducibility: We commit to open-sourcing our complete backtesting environment and RL training framework to enable full reproducibility. We will protect only the specific feature engineering components that provide competitive advantage, ensuring the core methodology remains accessible while balancing academic contribution with practical deployment considerations.

Technical Details on Hedger Pool Construction

Comprehensive details are provided in Appendix B.2, including:

• Selection criteria: Top-30 performers based on validation performance across market regimes
• Hyperparameter ranges: Tables 5-6 detail all parameters for Delta-based, Price-based, and Deep Hedgers
• Training methodology: Different risk-aversion levels (λ from 0.1 to 800) ensure diverse hedging behaviors

HR-Agent Responsiveness in Real-time Trading

Training vs. deployment distinction: During training, the HR-Agent switches hedgers at fixed 24-hour intervals (nhr = 24) to ensure RL training stability - frequent switching would cause training instability and convergence difficulties. However, in live deployment, the HR-Agent can operate in an event-driven manner, responding immediately to significant market moves or Greek threshold breaches. The fixed interval is a training constraint, not a deployment limitation.

Generalizability to Different Markets

Our focus on cryptocurrency options markets is driven by practical advantages and research value:

Data accessibility: Crypto derivatives provide comprehensive 24/7 data access essential for robust RL training, unlike traditional options markets with limited historical data availability.

Research value: As an emerging market, crypto options offer higher research value and cleaner microstructure without complex market maker intermediation, allowing focus on core volatility trading dynamics.

Framework generalizability: Our RL methodology can extend to different markets, with success primarily depending on market-specific feature engineering. Even within our study, BTC and ETH required different feature engineering approaches to achieve optimal results (though all baselines for each asset used identical features). However, feature engineering optimization falls outside this research's scope - our contribution lies in the RL framework design rather than market-specific feature selection.

Our research aims to demonstrate that RL methods indeed have advantages over traditional approaches in the volatility trading domain.

Additional Details on Advanced Volatility Model Baselines

As mentioned in our earlier response, we implemented and evaluated volatility forecasting baselines using both GARCH and deep volatility forecasting networks for realized volatility prediction. Below, we provide additional details regarding their implementation.

• GARCH: We implemented the standard GARCH model, trained on rolling daily returns using a one-year window. It was used to predict the realized volatility for the following 24-hour horizon.
• DeepVol: We implemented the DeepVol architecture proposed by Moreno-Pino and Zohren (2024) [1], which employs dilated causal convolutions to capture long-range dependencies from 5-minute return sequences. The model was trained using a one-year rolling window and used to predict the realized volatility for the following 24-hour period.

To ensure consistency across experiments, we employed the same decision-making logic as used in other baselines, conducted hyperparameter tuning for each model, and evaluated both GARCH and DeepVol using the same delta-based hedger in other baselines.

We hope this additional details provides a clearer understanding of our supplementary experimental setup and shows the inherent difficulty in converting volatility forecasts into trading gains. We remain happy to address any further questions you may have.

[1] Fernando Moreno-Pino and Stefan Zohren. DeepVol: Volatility Forecasting from High-Frequency Data with Dilated Causal Convolutions

Thank you for your thorough and insightful review. Please find our responses below.

Omission of Transaction Costs

We apologize for not clearly stating this in the paper. We do consider transaction costs in our simulation, which are deducted from the cash balance when trades are executed. Our implementation follows Deribit's fee structure: Commission fees: 0.05% for perpetual futures and 0.03% for options (capped at 12.5% of option price). We also consider the Bid-ask spread costs: We use market orders in backtesting, so spread costs are implicitly included through realistic execution prices.

Importantly, our trading policy is not high-frequency. As shown in Table 3, average holding periods range from 9-21 hours for long positions and 47-51 hours for short positions. The HR-Agent selects hedgers every 24 hours (Table 6), making transaction costs a relatively small component of total P&L.

The following table shows transaction costs as a percentage of total P&L in our experiments:

We will add this crucial information to the revised manuscript.

Unrealistic Assumption for Greek Calculation & Advanced Vol Smile Models

We acknowledge that BSM makes simplifying assumptions, but our framework is designed to be robust to Greek calculation imprecision for several key reasons:

1. Greeks as reference values, not precise inputs: Our method doesn't rely on perfect Greek precision but uses them as reference signals. BSM remains widely used by market participants despite its limitations - experienced traders understand these constraints and adapt accordingly.

2. Data-driven error correction: The Deep Hedgers take Greeks as input features, and the HR-Agent routes among hedgers based on their actual performance during RL training. This means model misspecification is automatically "priced in" through the data-driven learning process, similar to how human traders compensate for known BSM limitations.

3. Residual PnL explanation: The high residuals in Table 13 are primarily due to discrete time intervals (5-minute data) rather than Greek misspecification. Second-level data would yield much smaller residuals, as the Taylor expansion in Eq. (2) assumes continuous hedging. Our PnL decomposition serves to qualitatively analyze whether our strategy's profit sources align with expectations (short positions earning theta, long positions earning gamma).

Why advanced smile models aren't necessary: More sophisticated volatility smile models (e.g., SABR) focus on precise pricing across the entire volatility surface, typically used for market-making or volatility surface arbitrage. Our strategy trades only ATM straddles with relatively low frequency, making precise smile modeling unnecessary for our specific application.

Why ML-based Strategies Fail

The poor performance of ML baselines despite good forecasting metrics (Table 10 vs Table 2) highlights fundamental challenges in translating predictions to profitable trading:

1. Model assumption burden: ML forecasting requires explicit volatility modeling choices (estimators, frequencies, etc.) that introduce model misspecification risks. RL circumvents these assumptions through direct action optimization, adapting to market dynamics that traditional volatility frameworks may not capture.

2. Forecasting-execution gap: Building on these modeling limitations, ML models excel at volatility forecasting but struggle with the sequential, path-dependent execution required for options trading. As shown in Section 2.2 and Figure 1, optimal hedging decisions depend on complex interactions between market conditions, position Greeks, and future price paths. Converting point forecasts into profitable trading sequences requires manually designed rules that are difficult to optimize across market regimes - precisely the challenge our grid search experiments revealed.

These fundamental issues (1 & 2) affect all prediction-based approaches, while ML methods also face an inherent limitation:

3. Fat-tail prediction failure: While ML models achieve good overall forecasting performance, Table 11 shows they perform poorly on extreme volatility events (top 5% outliers) where most long-gamma profits are generated. RL's exploration and adaptation capabilities provide advantages precisely in these critical profit-generating scenarios.

End-to-end ML forecasting baseline: We add an end-to-end ML baseline that circumvents problems 1 & 2 following [1]. Please refer to the detailed experiment results in section Additional baseline details (DLOT) in our response to Reviewer JxTL. Results show that supervised learning methods still have significant disadvantages compared to RL in this scenario, demonstrating the superiority of RL's adaptive learning approach even when modeling assumptions are minimized.

Oracle Policy Sensitivity Analysis

The Oracle policy serves as a crucial bootstrap mechanism during Phase 1. To ensure diverse and profitable experience generation, we adopt a range of β thresholds when constructing the Oracle policy. Specifically, as shown in Table 4, we vary the Oracle threshold β across multiple values (0.1, 0.2, 0.4, 0.6, 0.8) to produce a broad spectrum of trading signals. This diversity encourages the OP-Agent to learn a richer set of profitable behaviors during pre-training.

These varied Oracle signals are then incorporated and refined in Phase 2, where the agent learns to adaptively combine and build upon these strategies. This two-phase approach is critical due to the delayed nature of profit realization in volatility trading, providing structured guidance toward profitability early in training.

Based on our design rationale, the Oracle need not be optimal - it simply needs to demonstrate profitable trading patterns that reduce exploration burden. Shortening or skipping the pre-training phase would lead to significantly slower convergence, as the agent would lack initial guidance and struggle to discover effective volatility trading strategies through exploration alone amid market noise. We will incorporate these explanations into the revised manuscript for clarity.

Reference:

[1] Deep Learning for Options Trading: An End-To-End Approach. ICAIF 2024.

Thank you for your valuable comments and positive assessment. Please find our detailed responses below.

Why RL is the right solution

Traditional approaches that rely on explicit volatility forecasting followed by rule-based trading decisions face fundamental limitations that make RL particularly well-suited for volatility trading:

1. Minimizing model assumptions: Traditional methods depend heavily on explicit modeling assumptions - IV calculations rely on option pricing models such as BSM, while RV estimation requires choosing among different variance estimators (maximum likelihood, maximum posterior, different data frequencies, etc.). In practice, selecting the appropriate models and parameters across varying market conditions is extremely challenging and often leads to model misspecification. RL circumvents these issues through data-driven learning that incorporates all market dynamics directly into the reward signal, eliminating the need for multiple separate models to estimate volatility components.
2. Sequential decision-making & Path-dependent profit: Volatility trading presents inherently sequential challenges where current decisions affect future risk exposures and hedging costs/timing - this is not a static optimization problem. As shown in Section 2.2 and Figure 1, optimal hedging decisions depend on complex interactions between market conditions, position Greeks, and future price paths. Current option positions determine future delta exposure, which influences subsequent hedging requirements and costs. In our experiments, we conducted extensive grid search optimization for rule-based strategies but still struggled to find configurations that generate stable profits across different market regimes. Traditional approaches using predictions with manually designed rules are prone to getting trapped in local optima or failing entirely. RL naturally handles these sequential dependencies by learning optimal action sequences that account for future implications of current decisions.

Intuition of the two-phase approach

Oracle policy initialization has been used in trading RL [1,2], but volatility trading presents unique challenges that motivate our two-phase design. The reward structure is highly path-dependent, and profitable trading trajectories are relatively sparse - making it extremely difficult to discover profitable strategies through random exploration alone. Moreover, the few profitable trajectories can be easily overwhelmed by noisy market data, leading to poor convergence.

Our Oracle policy uses delta-hedged P&L signals to guide the agent toward profitable trading behaviors, significantly reducing exploration costs. This initialization ensures the agent starts from a reasonable region of the policy space, then RL training further optimizes beyond this foundation to discover superior strategies that pure exploration would struggle to find amid market noise.

Deep Hedger reproduction

Yes, we reproduce the Deep Hedger following [3]. Appendix B.2 describes the details of different hedgers including Deep Hedgers, and Table 5 shows the parameters used for these hedgers. Our implementation follows the actor-critic framework with risk-adjusted utility functions and varying risk-aversion parameters as described in the original work.

Performance Guarantee

Theoretical considerations: While we don't provide formal convergence guarantees as this is an empirical RL research, in our research practice we found the following elements crucial for performance assurance:

1. Oracle policy initialization: The Oracle policy in Phase 1 ensures we start from a profitable region of the policy space, reducing the risk of converging to poor local optima.
2. n-step TD learning: Effectively reduces noise in Q-value estimation, making RL training convergence more stable compared to 1-step methods.
3. Lower-frequency HR-Agent switching: The HR-Agent's less frequent hedger selection (every 24 hours) similarly improves training stability by reducing action space complexity.
4. Relative reward design for HR-Agent: Using the advantage of selected hedgers over baseline hedgers as reward provides more stable learning signals than absolute performance measures.

TWAP/VWAP baselines

These execution algorithms are designed for minimizing market impact when trading large orders, which is orthogonal to our volatility trading objective. Our baselines already comprehensively cover the relevant comparison space: directional strategies (Long/Short), factor models (MR/MOM), and ML forecasting approaches (GBDT/MLP/LSTM). TWAP/VWAP would not provide meaningful comparisons for volatility timing and Greek management, as they address fundamentally different trading problems.

References:

[1] Universal Trading for Order Execution with Oracle Policy Distillation, AAAI 2021 [2] EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading, AAAI 2024 [3] Deep Hedging: Continuous Reinforce359 ment Learning for Hedging of General Portfolios across Multiple Risk Aversions, ICAIF, 2022

I put in the justification that "Thanks for the authors' responses. They address my concerns well. And I will keep my scores."

One quick question: Are the datasets and implementation codes publicly available? If not, will them be released in the future?

There is no more concern from my end. The rest is left to the meta-reviewer and AC. Good luck!