FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making

[Q2: Optimization approach] We use an RL-inspired Actor-Critic structure to dynamically update manager agent's belief other than implementing a classic RL.

We do not employ RL optimization directly. Our work models financial trading tasks as a POMDP and addresses the optimization problem using a textual gradient approach. We incorporate the Actor-Critic (AC) structure from traditional reinforcement learning due to its broad applicability for designing intelligent agents. However, our CVR algorithm, which integrates textual gradient descent, the AC structure, and quantitative risk management, substantially differs from the traditional AC RL algorithm. Additionally, unlike traditional RL algorithms, we do not modify the intrinsic parameters of the LLMs.

[Q3: Implementation details of CVR] We have provided more details about the CVR percentage overlap across iterations. For more technical details, please request our submitted code from the AC.

In the training stage, the CVR algorithm evaluates and adjusts optimization process across training episodes by analyzing conceptualized investment insights. Overlapping percentages of insights between episodes act like a learning rate, guiding necessary adjustments: A high overlap suggests minor tweaks are sufficient, while a low overlap signals the need for more significant changes to improve performance. In our experiment, a 50% overlap between the 1st and 2nd episodes indicates significant changes are needed, while subsequent increases to 62.5% and 72.5% suggest progressively finer adjustments. This pattern shows how the algorithm adapts and refines strategies, achieving stability and improved performance through iterative learning.

[Q4:Trading volume integration] Trading volume is factored into each decision-making process, as elaborated below.

In our system, stock selection agents analyze multimodal market data to curate a stock pool based on statistical correlations between stock returns. The manager agent assigns long, short, and neutral positions to each stock, which act as constraints for the mean-variance optimization performed daily to determine portfolio weights. These weights are then linearly scaled to define target positions (scaled between 0 and 1). Finally, our back-testing system computes the number of shares to buy or sell for each asset based on its allocated buying power and current price. It is to be noticed that, though experimenting on a compact portfolio due to budget limitation, our working mechanism can be easily generalized to larger-size portfolios and is the first to employ LLM agents for managing a portfolio, marking a significant innovation in financial decision-making.

[Q5: Experimental results] We updated results with requested information.

Please refer to [W1] for detailed discussions. Please see the updated results in Table A in [W1].

[Q6:Manual labor] Our prompt engineering utilizes automated generation, not solely manual labor.

The creation of ticker-specialized prompts for the profile module (Appendix J) employs an automated process. We developed a uniform prompt template that is automatically adapted for different stock tickers, significantly reducing manual effort. With approximately 3,000 actively traded US stocks, this approach is manageable with our current resources. Future implementations will incorporate advanced automated prompt-generation techniques, further minimizing manual input and enhancing prompt customization efficiency.

[W3: Contribution of each FINCON component/ Ablation of memory module.] We have added an ablation study to assess the effectiveness of FINCON's episodic memory design.

1. To clarify the contribution of each part, we provide our detailed motivation as follows:

• Multi-agent setup: We employ a unique Manager-Analyst hierarchical multi-agent structure, where each agent is responsible for a specific task. This synthesized approach enables collaborative tackling of sequential decision-making challenges in complex financial markets. The hierarchical communication structure enhances efficiency by eliminating redundant peer-to-peer interactions, thereby optimizing resource utilization.

• Reinforcement learning: The actor-critic structure in classical RL serves as an inspiration for our system design, though FinCon adapts its core principles rather than applying them directly. The actor component in our system is tasked with selecting policies. The critic component in our system does not estimate value functions as in traditional RL,it is designed to reflect on and learn from past actions by evaluating their outcomes based on feedback from the environment.

• Conceptual Verbal Reinforcement (CVR): The CVR helps to update manager's investment beliefs over multiple training episodes. The ablation study in [W3.2] below further illustrates its contribution to FINCON's performance.

• Convex optimization: Convex optimization is employed exclusively to determine asset allocation in portfolio management tasks, and we leverage this method for its proven reliability and effectiveness in portfolio management. Our work is pioneering in its integration of LLMs for managing portfolios, marking the first instance of such an application in the field.

These mechanisms work in concert to ensure FINCON's high-quality decision-making capability from different perspectives. Furthermore, we believe the setup of FINCON maintains novelty from two fronts: Empirical novelty: FINCON addresses the complexity of volatile market dynamics and achieves state-of-the-art performance in various practical financial decision-making tasks. Moreover, it is the first financial language agent system to support portfolio management functionality. Technical novelty: FINCON introduces the unique CVR mechanism for multi-agent system risk control. This represents a novel development beyond verbal reinforcement, designed specifically to synthesize hierarchical multi-agent collaboration.

2. Our memory module is composed of working memory, procedural memory, and episodic memory. While the first two components support essential functions for each agent's basic operations, the episodic memory refers to the updated investment beliefs through FINCON's CVR mechanism, continuously enhancing its decision-making capabilities over episodes. Therefore, conducting an ablation study on episodic memory is more meaningful. Our experimental outcomes are summarized in the same Table C in W2. Given the same choice of training and test periods as Table A and B, the results demonstrate that CRs and SRs have been substantially enhanced after iterating over four training episodes of investment belief updates. This conclusion holds true for both single stock trading and portfolio management tasks, and remains robust across various market conditions (bearish, bullish, and mixed).

[W4: Reproducibility] We have shared the source code of FinCon with standard protocol through AC.

Please don't hesitate to request it from the AC.

[W5: Citation] We will add this paper as part of our related work and citation list.

We agree the significant position of this paper in the field of textual optimization. We will incorporate it in the related work. And, the paper will also be properly cited in our reference list.

[Q1: CVR vs. OPRO] Differences Between CVR and OPRO are explained below.

1. OPRO includes the entire optimization trajectory in meta-prompts, while CVR selects segments of consistent profit or loss.
2. Unlike OPRO, which only specifies the upgrade direction, CVR mimics the learning rate using the overlapping rate between conceptualized insights.
3. OPRO handles deterministic problems like linear regression and the traveling salesman problem, as well as manually annotated datasets (e.g., GSM8K [1], BBH [2]). In contrast, CVR addresses stochastic optimization in financial investment, managing the environment's randomness and complexity.

[1] Cobbe, Karl, et al. "Training verifiers to solve math word problems." arXiv preprint arXiv:2110.14168 (2021).

[2] Suzgun, Mirac, et al. "Challenging big-bench tasks and whether chain-of-thought can solve them." arXiv preprint arXiv:2210.09261 (2022).

We appreciate the reviewer's feedback. In response to the identified weaknesses and limitations:

[W1: Experimental results] We have updated the experimental results to include other three stocks.

We have updated the experimental results to include other three stocks including AAPL, NIO and AMZN. And the new results still suggest that our model still significantly outperforms the comparative models regarding to key metrics, Cumulative Return (CR) and Sharpe Ratio (SR), as depicted in Table A.

Table A: Comparative analysis of trading agent systems on single asset task: FINCON outperforms on key metrics (Cumulative Returns (CRs) and Sharpe Ratios (SRs)) across multiple stocks. As the COIN first IPOed in 2021, the RL algorithms failed to converge to a stable result with limited data. Thus, We marked the metrics as 'NA'. Note: Due to the space limit, we only include the values of primary metrics.

[W2: Stock selection and portfolio construction/ Ablation about CVR] We have clarified the criteria to select the stock and added experimental results for another portfolio. An ablation study for the CVR mechanism has also been added.]

1. Other than simply enumerating all combinations of 3 tickers out of 5 available, our stock selection agent within FINCON constructs the portfolio based on the classic financial principle of diversification from a pool of 42 tickers (number determined by data APIs accessible to us), which achieved by constraining the statistical correlation between the candidates in this pool. Fixing a stock as the target, we found the tickers maintain the most diverse historical return patterns compared to it to form the portfolio.At the same time, we eliminated the combination with tickers has less than 1000 news records in the running period. Additionally, we had the stock selection agent execute another round of portfolio construction with AMZN as the target, resulting in a new portfolio: AMZN, GM, and LLY. FINCON again exhibited significantly better performance over comparative models in terms of CR and SR, further illustrating its robustness in portfolio management tasks (Table B below). While we currently use FINCON to trade compact portfolios (three symbols) due to time and budget constraints, its promising performance suggests a potential for managing larger portfolios.

Table B: Key performance metric comparison among all management strategies for a portfolio consisting of (AMZN, GM, LLY). All baselines are kept the same as the ones in the paper.

2. In response to the ablation study about CVR, we have updated the results in Table C, as well as in [W3] for a more detailed discussion. In the table, we present the results with and without CVR updates for both single stock trading and portfolio management tasks. The results demonstrate that implementing the CVR mechanism significantly improves decision-making quality across various tasks and market conditions.

Table C: Key metrics FINCON with vs. without implementing Conceptual Verbal Reinforcement（CVR）/ investment belief updates for over-trajectory risk control. The performance of FINCON with the implementation of CVaR won a leading performance in both single-asset trading and portfolio management tasks.

Thank you again for your feedback.

We have added the necessary material to the author rebuttal and official comments to address the points you raised. Given that we only have three reviewers, each reviewer's score significantly impacts the overall assessment of our work. We believe that the current overall assessment does not adequately reflect the contribution of our work. Therefore, we kindly request you reconsider your score. Thank you again for your time and effort in reviewing our paper.

[Q1： MSFT results] - Sure!: We provided MSFT result in below.

The reason we did not add the MSFT is that we missed this stock in the updated result. Here are the updated results for MSFT. For further explanation on performance, please refer to our answer to Q2.

Although FINCON marginally outperforms the buy-and-hold strategy in a bullish market condition, where a naive buy strategy would be advantageous, our model gains a clear advantage over other autonomous trading systems.

[Q2: Clarification on our model results]- We have updated the experiment with extended training and test periods.

The variation in evaluation metrics for TSLA, NFLX, and COIN between Table A in the rebuttal and Table 1 in the original manuscript is not due to chance but because of the following two reasons:

1. Extended data collection:

Initially, our training period spanned from January 2022 to August 2022, and we used testing data through April 25, 2023, as shown in Figure 3 of the original manuscript. As part of this ongoing project, we have continued to expand our data collection to include the most recent news data, updating our dataset through June 10, 2023, during the rebuttal period. This expansion has enabled us to extend the training period from January 2022 to September 2022. This longer warm-up period enabled our manager agents to receive more extensive updates on investment beliefs via CVR. Additionally, we retain a longer and more up-to-date test period (from the beginning of October 2022 to June 10th, 2023) to examine our agent performance. Our rebuttal results, including all baselines, are conducted in these same training and testing time periods.

Additionally, we would like to make further clarifications for our submitted paper and first rebuttal reply: Cumulative returns (CR%) should be reported in Table 1 of the original manuscript in percentage. We input them as decimals for MSFT, AMZN, NFLX, and COIN as decimals by dismissal. This error also affected the CRs for NFLX in Table A of W1 in our rebuttal reply. We have fixed this issue and included the correct results in the following Updated Table A.

Updated Table A: Comparative analysis of trading agent systems on single asset task: FINCON outperforms on key metrics (Cumulative Returns (CRs) and Sharpe Ratios (SRs)) across multiple stocks. As the COIN first IPOed in 2021, the RL algorithms failed to converge to a stable result with limited data. Thus, We marked the metrics as 'NA'. Note: Due to the space limit, we only include the values of primary metrics.

Another thing that I’d like to bring up is about your answer to “7. Experiment Statistical Significance” in the NeurIPS checklist. Speaking about Table 1, the confidence intervals are not provided, thus the answer to the questionnaire must be No. Looking at how the CR numbers vary between the original manuscript and the updated table, the CR numbers exhibit significant variation. Even the new test period size is small and can hardly be increased for the time series problems. A convincing analysis of the distribution of CR outcomes could have been done either with re-running the experiments multiple times with significant LLM temperature, or by re-running on different slices of your total data. For the latter, the example could be:

Fold 1: 0-60% train, 60-90% test

Fold 2: 1-61% train, 61-91% test

…

Fold 10: 10-70% train, 70-100% test.

Either of these will produce the standard deviation of CR outcomes needed to estimate the statistical significance of the results.

Anyway, even without checking the statistical significance, the current results are good. I plan to update my judgment to Accept.

We deeply appreciate your positive feedback on our submission and your willingness to consider updating the rating to acceptance level.

Thank you for your insightful review and the suggestions you’ve made, which have improved the manuscript.

Your point about the original vs. final manuscript is understandable: the way we see, the reviews prompted us to dig deeper on some of the questions raised, and we think inclusion of what we found in the revision is worthwhile. Your suggestion about the confidence intervals is a good one, and we will rerun our experiments based on your suggested settings and incorporate these metrics in the further revision. And we will also update the checklist.

As the discussion deadline approaches, shall we kindly ask that you adjust the rating score to reflect your consideration of acceptance? We appreciate your support. Thanks again.

We thank the reviewer for your appreciation of our work, particularly regarding the innovative nature and efficacy of the FINCON framework. The reviewer's concerns about FINCON's weaknesses and limitations are carefully explained and addressed below.

[Weakness: Transparency] The multi-agent interactions are transparent, and the risk-control mechanism is fully accessible.

1. The multi-agent interactions are conveyed and recorded by natural language, which is completely accessible to users. This fosters trust compared to other autonomous trading systems. This makes them suitable for monitoring and intervention if necessary. Additionally, the memories stored in the memory module can be queried in a structured form for review when needed.

2. The FINCON system features a dual-level risk control mechanism designed to enhance decision-making quality in volatile market environments while ensuring full transparency: (1) In-trajectory risk control, which monitors for sudden market risks during an ongoing episode. This mechanism is activated when there is a decrease in Conditional Value at Risk (CVaR), signaling that recent trading decisions might have pushed Profit and Loss (PnL) values into the worst 1% of outcomes, thus indicating high-risk conditions; and (2) Across-trajectory risk control, which refines the manager agent's investment strategies over multiple episodes. This is done by optimizing the integration of analyst-provided information for specific trading objectives. Both levels of risk control are operationalized through natural language prompt templates directed at the FINCON manager agent. The in-trajectory risk alerts are conveyed through real-time LLM prompts, while the across-trajectory adjustments are implemented by periodically updating the manager's profile prompts with new investment beliefs every two episodes. This structure ensures transparency of its internal mechanism and allows users to understand and potentially tailor the risk management strategies employed.

However, the intrinsic mechanisms of LLM are still under active research [1]. We will add a discussion about the limitation in the future revision.

[1] Luo, H., & Specia, L. (2024). From understanding to utilization: A survey on explainability for large language models. arXiv preprint arXiv:2401.12874.

[Limitation: Scalability and computational demand] Scalability and computational demand will not be a bottleneck in our case.

We appreciate the reviewer's inquiry. Our work is the first to use LLM agents for portfolio management, representing a significant innovation in AI-driven financial decision-making. While real-time processing may seem time-consuming, we operate at a daily trading frequency, a common lower-frequency approach in real markets. In our case, each decision-making step around 20 seconds to 1 minutes using an agent backed by LLM, such as GPT-4 Turbo, which is well-suited for daily trading. Given the comparison between the required trading frequency and our current average response time, FINCON demonstrates significant potential to manage larger-scale data streams and portfolios effectively, as long as the data is accessible. In our future research, we plan to incrementally test the model on larger and more complex datasets, and we also intend to incorporate a wider variety of financial instruments, such as options, futures and swaptions.

Table A: Comparative analysis of trading agent systems on single asset task: FINCON outperforms on key metrics (Cumulative Returns (CRs) and Sharpe Ratios (SRs)) across multiple stocks. As the COIN first IPOed in 2021, the RL algorithms failed to converge to a stable result with limited data. Thus, We marked the metrics as 'NA'. Note: Due to the space limit, we only include the values of primary metrics.

Table B: Key performance metric comparison among all management strategies for an additional portfolio consisting of (AMZN, GM, LLY). All baselines are kept the same as the ones in the paper.

Table C: Key metrics FINCON with vs. without implementing Conceptual Verbal Reinforcement（CVR）/ investment belief updates for over-trajectory risk control. The performance of FINCON with the implementation of CVaR won a leading performance in both single-asset trading and portfolio management tasks.

We thank the reviewer for recognizing the novelty and soundness of our approach. To address your mentioned weakness and questions:

[W: Ethical concerns and market impact] Clarification about Ethical Concerns & Impact on Market Dynamics

1. Regarding Data:

We fully respect the copyright of data sources and adhere to all relevant guidelines. For public datasets, there are no concerns regarding copyright or other restrictions. The datasets we used, such as Form 10-K, Form 10-Q, and earnings conference calls, are required filings with the SEC and are archived in publicly accessible databases. For proprietary datasets, we respect their copyrights and have retrieved them from valid sources with proper consent. Although we did not disclose the data to comply with copyright requirements, we have clearly specified their sources and provided the necessary code for future researchers to reproduce our work. In addition, we plan to include copyright information of our data source in Appendix B in future revisions.

2. Regarding Market Impact

FinCon is primarily an academic exchange product and is not intended for commercial use. We will emphasize this point on our homepage, in articles, and in our code. However, if the industry is inspired by our work to design similar trading systems, we still have reason to believe that this would be beneficial for financial markets. Such a system is an automated trading system that accelerates the reflection of public information in the market, helping stock prices to converge to their true value and benefiting investors [1]. This ensures that they can trade the underlying asset at a fair price and reduce their losses due to delayed price updates. It aligns with the classical Efficient-Market Hypothesis in finance, which states that prices should immediately reflect all available information [2]. Moreover, as an assistant, the system can reduce human errors, thereby enhancing accuracy and reliability in financial operations [3]. This, in turn, decreases systemic risks in financial markets and improves the overall efficiency of financial services.

By incorporating the mentioned improvements, we believe we can fully address ethical concerns raised by reviewers, and we think the development of such an automated trading system can benefit the financial market.

[1] Dubey, R. K., Babu, A. S., Jha, R. R., & Varma, U. (2022). Algorithmic trading efficiency and its impact on market-quality. Asia-Pacific Financial Markets, 29(3), 381-409.

[2] Miller, C. N., Roll, R., & Taylor, W. (1970). Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25(2), 383-417.

[3] Todorović, V., Pešterac, A., & Tomić, N. (2019). The impact of automated trading systems on financial market stability. Facta Universitatis, Series: Economics and Organization, 255-268.

[Q1: Robustness] Generalization to Multiple Financial-decision Marking Tasks

We have conducted extensive experiments and ablation studies to demonstrate FINCON's robustness in decision-making tasks, particularly in single-stock trading and portfolio management. For more details, please check our latest results in Table A, B, and C comment box below and the Table 1, 2, 3 and 4 in the pdf file attached with author rebuttal. Our evaluation spans multiple assets under diverse market conditions. Notably, FINCON exhibits strong generalization capabilities in making automated investment decisions for both individual and combined financial products. This versatility indicates that FINCON's trading objectives could potentially extend to other financial decision-making tasks, including investments in cryptocurrencies, ETFs, or combinations thereof, provided relevant data is available. We will further add a discussion about FINCON's other potentials in our future revisions.

[Q2: Risk-control component working mechanism] FINCON's risk-control component directly impacts the manager's reasoning and influences analyst outcomes through the manager agent's feedback, which is informed by the present risk scenario.

FINCON provides a two-fold risk control mechanism to dynamically update part of the manager's decision-making contexts: (1) in-trajectory risk control alerts the manager agent to emerging market risks during active episodes, enabling immediate strategy adjustments; and (2) across-trajectory risk control adjusts the manager agent's investment beliefs for better weight allocation of information perspectives provided by each analyst. The belief updates focus on certain trading targets and is refined over episodes. In summary, our risk-control mechanism directly and timely shapes the manager agent's critical investment-related contexts in the decision-making process. The manager agent, considering these contexts, identifies critical memory insights that facilitate its decisions and informs analyst agents accordingly. The analyst agents then reinforce these memories in their memory system by increasing their ranking scores, ensuring that information considered useful for decisions has a higher chance of being selected for future decision-making and a slower decay rate from the system. Through this process, the risk-control component optimizes the workflows for both manager and analyst agents as a synthesized system.

We collected performance metrics for both single stock trading (TSLA) and portfolio management (combination of TSLA, MSFT, and PFE) tasks, focusing on Cumulative Returns (CRs) and Sharpe Ratios (SRs). The data covers a training period from 2022-01-17 to 2022-03-31 and a test period from 2022-04-01 to 2022-10-15. We selected this time range because the VIX (CBOE Volatility Index) maintained a high level (averaging above 20) during this period, indicating greater market volatility than usual.

Below, we summarize the top 3 best-performing models for single stock trading in Table D below. FINCON is the only agent system achieving positive CRs and SRs for single stock trading tasks. For the complete results of all baselines, please refer to Table 2 in the PDF attached to the author rebuttal. For the portfolio management task, all baseline results (four benchmarks) are provided in Table E, in which the FINCON attained the highest values in the primary performance metrics.

These empirical outcomes support FINCON's robustness during periods of significant market volatility.

Single Stock: TSLA

Table D: Top 3 best-performing models for single stock trading (TSLA) ranked by Sharpe Ratio (SR) under the high volatility condition. FinCon demonstrated the best performance for key metrics. Please refer to Table 2 in the PDF of the author rebuttal to see the full results for all baselines.

Portfolio:

Table E: Key performance comparison among all portfolio management strategies under the high volatility condition. The portfolio consists of (TSLA, MSFT, PFE). All baselines are kept the same as the ones in the paper.