SocioDojo: Building Lifelong Analytical Agents with Real-world Text and Time Series

We deeply appreciate your insightful feedback and recognition of the significance of our work, SocioDojo. We have taken your comments into careful consideration and offer the following detailed responses:

For the Weaknesses:

1. While the SocioDojo environment is comprehensive and emulates the running of the world, its complexity might make it challenging for researchers to quickly adapt.

We acknowledge that the comprehensive nature of SocioDojo presents a learning curve. To provide some help to assist researchers in navigating this complexity, we have included a new figure in Appendix D. This figure provides an overview of the collaborative decision-making process involving the Analyst, Assistant, and Actuator, offering a clearer understanding of their roles and interactions within SocioDojo.

2. I think it would be beneficial to introduce some actual real-world classic scenarios or investment cases. These could serve as short-term goals or benchmarks, allowing researchers to run and evaluate their own model in a phased manner. It could also be useful for evaluating the information coverage level of the system.

Introducing real-world scenarios and investment cases is indeed a valuable suggestion. We plan to develop and include these examples in SocioDojo to serve as benchmarks and to enhance the system's evaluation in terms of information coverage and real-world applicability.

3. The lower bound of the system is not clear. The complexity of SocioDojo might inadvertently obscure the foundational or simpler strategies that also be effective. For instance, SOTA time-series forecasting methods could be employed as a foundational strategy for investment within the SocioDojo environment. It could serve as a baseline or a lower bound against which more complex strategies can be compared.

While SOTA domain-specific models in areas like quant finance could provide a foundational strategy within SocioDojo, our focus is on creating fair baselines with autonomous agents geared toward general socio-intelligence. However, incorporating expert domain models and tools into the agent's analysis process presents an exciting future direction, leveraging the capabilities of large language models (LLMs) for tool utilization.

4. I also hope the author put more discussion on how the authors have addressed potential biases, screened sources, and ensured the diversity of data, it would bolster the paper's credibility and address potential concerns.

To address potential biases and source diversity concerns, we have added Appendix B, detailing our methodologies for screening sources and ensuring data diversity.

For your Questions:

1. Are there any data leakage between the knowledge base and the streaming messages?

To prevent data leakage, we have strictly used only materials published before the start date in the knowledge base.

---

We hope these responses comprehensively address your points and enhance your understanding of our project. We greatly appreciate your constructive feedback and remain open to any further inquiries or suggestions for improvement. Thank you for your support and valuable contribution to our work's development.

We greatly appreciate your thoughtful feedback and acknowledgment of our work on SocioDojo. After careful consideration of your comments, we offer the following responses:

For the Weaknesses:

1. Since GPT-3.5-Turbo was used as foundation models with a non-zero (0.2) temperature, the results should not be fully deterministic, while the paper may miss some studies on randomness.

We choose a non-zero setting as 0 temperature may restrict the model performance. While the chosen temperature of 0.2 (on a scale of 0 to 2) is relatively low and aligns with recommendations for deterministic tasks, we recognize the value of further exploring how varying temperatures might affect outcomes. We are currently conducting additional studies on this aspect and plan to include these findings in future updates of our work.

2. The setting of "Forbid day trading", that an asset cannot be sold within 5 days of purchase to avoid profiting from short-term patterns, might be overly strict.

The stringent "Forbid day trading" rule was designed to elevate the cost of decision-making, thereby promoting depth and consideration in the agents' analyses. This aligns with our goal of fostering high-quality, thoughtful decision-making. Additionally, our use of daily close price data, which are public while minute-level data are usually commercial, necessitates such a restriction to counterbalance the potential information advantage that agents might have due to data delays. We have added Appendix D to examine the implications of utilizing close price data.

3. For the experimental setting of "tracking", it remains unclear how the portfolio performs against an actual index tracker.

In response to your query about portfolio performance in the "tracking" experimental setting, we have added a comparison with major indices like S&P 500 and FANG+ in Appendix C. This comparison helps contextualize our agent's performance, although it's important to note that SocioDojo has not been explicitly optimized for financial market environments. The direct comparison may not be fair considering the agent's unique advantages and disadvantages, such as not adhering to trading hours and not utilizing real-time data, along with the restriction against day trading.

For your Questions:

1. When multiple news articles sent out on the same day, did the order of news matter? i.e., to which extend the agent is permutation invariant?

In SocioDojo, each news piece is timestamped to maintain chronological order, mirroring real-world scenarios. Overlaps in timestamps are rare, but even if they occur, they accurately reflect real-life information flows, ensuring that the agent's response is realistically grounded.

---

We hope these responses address your concerns and provide further clarity on our research. We are grateful for your feedback and remain open to any additional questions or suggestions you might have. Thank you once again for your valuable input and for the opportunity to enhance our work.

We are grateful for your comprehensive review and constructive feedback on our work with SocioDojo. Below, we address each of your concerns and questions:

For the Weaknesses:

1. The definition of POMDP seems to disconnect from the rest part of the manuscript. Those notations are not used in other parts at all.

We appreciate your observation regarding the use of POMDP in our manuscript. In Section 3.2, we clarify how SocioDojo implements the POMDP environment, with information sources forming the observation space $\Omega$, time series approximating the world state $S$, and the knowledge bases as context $Z$. This formulation in POMDP terms offers a structured and compact mathematical framework, enhancing the manuscript's comprehensibility and providing a common language for future research in this domain.

2. The main evaluation results are only based on a final return in the defined period, which misses other important aspects for the results to be valid. See questions below

Further addressed below in “Questions”.

3. The proposed “return” of “hyperportfolio”, as some kind of evaluation metric, is very hard to interpret. While it seems that it is related to how accurately the agent is able to predict the future of various time series, it’s hard to make sense of the numerical value, especially when various time series are fused together.

Further addressed below in “Questions”.

Dear Reviewer apFV,

We sincerely appreciate your detailed feedback and your reconsideration of our work. Regarding your concerns:

(1) if using existing forecasting is a legitimate strategy, then it's probably more reasonable to also include reliable existing forecasting as baselines.

We acknowledge there exist reliable domain-specific models in fields like quantitative economics and algorithmic trading. However, their focus differs from ours, our main focus is on general societal intelligence rather than domain-specific expertise. Such generality allows their applications to be extended beyond traditional time series forecasting such as business planning and social science research, as outlined in the introduction. Yet, directly competing with experts in all fields without domain-specific optimizations is not the target. We agree that comparison with domain experts can provide important insights into specific applications, but it will go beyond the main focus of this work.

(2) It remains a bit unclear how long would the agent hold for each time series, which might shed some light on the decision of the agent. For example, this holding period may reflect the frequency of public forecasting.

The holding period for each time series is dynamic adjustments with multiple buy/sell operations in different amounts at different time points. Thus, there is no determined holding period for a time series. We recognize the value of visualizing these holding periods can provide useful insights. Considering the rebuttal period is limited, we will incorporate this into future revisions.

(3) The return of hyperportfolio as an evaluation metric of an agent is not easy to interpret, especially with so many time series combined together. A positive return does not necessarily mean that a wise decision is made. Take GDP as an example, since a lot of countries have positive growth, it's fairly easy for an agent to invest in such a time series and obtain a reasonable positive return.

We have handled your concern on time series like GDP in Section 3.2.2: "Since some assets are almost ever-growing, such as GDP, which leads to risk-free returns, we wish all returns to come with risk, thus requiring good analysis to achieve a better risk-return trade-off." and further detailed in Appendix C Robustness Analysis of SocioDojo. We introduced mechanisms like overnight rate and explored the proper settings by experiment to eliminate "risk-free" returns for such time series to ensure every positive return can reflect an improvement in societal understanding and decision-making. We agree that developing more intuitive evaluation metrics for agents is an essential aspect of our future work, but also challenging akin to evaluating a fund in the real-world financial context.

---

We hope this response can address your concerns and aid in the further evaluation of our work. Your feedback is invaluable, and we remain open to any further questions or suggestions you may have.

Best regards,

Authors of Submission 2968