A Robust Concurrent Multi-Agent Deep Reinforcement Learning ‎based Stock Recommender System

Khonsha, S.; Sarram, M. A.; Sheikhpour, R.

doi:10.22061/jecei.2024.11193.775

Document Type : Original Research Paper

Authors

¹ Department of Computer Engineering, Zarghan Branch, Islamic Azad University, Zarghan, Iran.

² Computer Engineering Department, Yazd University, Yazd, Iran. ‎

³ Department of Computer Engineering, Faculty of Engineering, Ardakan University, P.O. Box 184, Ardakan, Iran.

https://doi.org/10.22061/jecei.2024.11193.775

Abstract

Background and Objectives: Stock recommender system (SRS) based on deep ‎reinforcement learning (DRL) has garnered significant attention within the ‎financial research community. A robust DRL agent aims to consistently ‎allocate some amount of cash to the combination of high-risk and low-risk ‎stocks with the ultimate objective of maximizing returns and balancing risk. ‎However, existing DRL-based SRSs focus on one or, at most, two sequential ‎trading agents that operate within the same or shared environment, and ‎often make mistakes in volatile or variable market conditions. In this paper, ‎a robust Concurrent Multiagent Deep Reinforcement Learning-based Stock ‎Recommender System (CMSRS) is proposed.‎
Methods: The proposed system introduces a multi-layered architecture that ‎includes feature extraction at the data layer to construct multiple trading ‎environments, so that different feed DRL agents would robustly recommend ‎assets for trading layer.‎‏ ‏The proposed CMSRS uses a variety of data sources, ‎including Google stock trends, fundamental data and technical indicators ‎along with historical price data, for the selection and recommendation ‎suitable stocks to buy or sell concurrently by multiple agents. To optimize ‎hyperparameters during the validation phase, we employ Sharpe ratio as a ‎risk adjusted return measure. Additionally, we address liquidity ‎requirements by defining a precise reward function that dynamically ‎manages cash reserves. We also penalize the model for failing to maintain a ‎reserve of cash.‎
Results: The empirical results on the real U.S. stock market data show the ‎superiority of our CMSRS, especially in volatile markets and out-of-sample ‎data.‎
Conclusion: The proposed CMSRS demonstrates significant advancements in ‎stock recommendation by effectively leveraging multiple trading agents and ‎diverse data sources. The empirical results underscore its robustness and ‎superior performance, particularly in volatile market conditions. This multi-‎layered approach not only optimizes returns but also efficiently manages ‎risks and liquidity, offering a compelling solution for dynamic and uncertain ‎financial environments. Future work could further refine the model's ‎adaptability to other market conditions and explore its applicability across ‎different asset classes.‎

Keywords

Main Subjects

Machine Learning

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0/

Publisher’s Note

JECEI Publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Publisher

Shahid Rajaee Teacher Training University

References

[1] M. Z. Asghar, F. Rahman, F. M. Kundi, S. Ahmad, "Development of stock market trend prediction system using multiple regression," Comput. Math. Organ. Theory, 25(2019): 271-301, 2019.

[2] A. A. Ariyo, A. O. Adewumi, C. K. Ayo, "Stock price prediction using the ARIMA model," in Proc. 2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation: 106-112, 2014.

[3] Y. Wang, Y. Liu, M. Wang, R. Liu, “LSTM model optimization on stock price forecasting,” in Proc. 2018 17th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES): 173-177, 2018.

[4] S. Banik, N. Sharma, M. Mangla, S. N. Mohanty, S. Shitharth, “LSTM based decision support system for swing trading in stock market,” Knowl. Based Syst., 239: 107994, 2022.

[5] S. Selvin, R. Vinayakumar, E. A. Gopalakrishnan, V. K. Menon, K. P. Soman, “Stock price prediction using LSTM, RNN and CNN-sliding window model,” in Proc. 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI): 1643-1647, 2017.

[6] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness et al., “Human-level control through deep reinforcement learning,” Nature, 518(7540): 529-533, 2015.

[7] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifreet al., “Mastering the game of Go with deep neural networks and tree search,” Nature, 529 (7587): 484-489, 2016.

[8] A. R. Azhikodan, A. G. Bhat, M. V. Jadhav, “Stock trading bot using deep reinforcement learning,” in Innovations in Computer Science and Engineering, Proc. the Fifth ICICSE 2017: 41-49, Springer, 2019.

[9] X. Wu, H. Chen, J. Wang, L. Troiano, V. Loia, H. Fujita, “ Adaptive stock trading strategies with deep reinforcement learning methods,” Inf. Sci., 538: 142-158, 2020.

[10] S. Carta, A. Corriga, A. Ferreira, A. S. Podda, D. R. Recupero, “A multi-layer and multi-ensemble stock trader using deep learning and deep reinforcement learning,” Appl. Intell., 51: 889-905, 2021.

[11] X. Y. Liu, H. Yang, Q. Chen, R. Zhang, L. Yang, B. Xiao, C. D. Wang, “FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance,” arXiv preprint arXiv:2011.09607, 2020.

[12] S. Yang, “Deep reinforcement learning for portfolio management,” Knowl. Based Syst., 278: 110905, 2023.

‎[13] C. Ma, J. Zhang, Z. Li, S. Xu, “Multi-agent deep reinforcement learning algorithm with trend consistency regularization for ‎portfolio management,” Neural Comput. Appl., 35(9): 6589-6601, 2023.‎

[14] Z. Huang, F. Tanaka, “MSPM: A modularized and scalable multi-agent reinforcement learning-based system for financial portfolio management,” Plos one, 17(2): e0263689, 2022.

[15] J. Lussange, I. Lazarevich, S. Bourgeois-Gironde, S. Palminteri, B. Gutkin, “Modelling stock markets by multi-agent reinforcement learning,” Comput. Econ., 57: 113-147, 2021.

[16] J. Lee, R. Kim, S. W. Yi, J. Kang, “MAPS: Multi-agent reinforcement learning-based portfolio management system,” arXiv preprint arXiv:2007.05402, 2020.

[17] P. Koratamaddi, K. Wadhwani, M. Gupta, D. S. G. Sanjeevi, “A multi-agent reinforcement learning approach for stock portfolio allocation,” in Proc. the 3rd ACM India Joint International Conference on Data Science & Management of Data (8th ACM IKDD CODS & 26th COMAD): 410-410, 2021.

‎[18] D. Kwak, S. Choi, W. Chang, “Self-attention based deep direct recurrent reinforcement learning with hybrid loss for trading signal ‎generation,” Inf. Sci., 623: 592-606, 2023.‎

[19] S. Forouzandeh, K. Berahmand, R. Sheikhpour, Y. Li, “A new method for recommendation based on embedding spectral clustering in heterogeneous networks (RESCHet),” Expert Syst. Appl., 231: 120699, 2023.‎

‎[20] S. Forouzandeh, M. Rostami, K. Berahmand, R. Sheikhpour, “Health-aware food recommendation system with dual ‎attention in heterogeneous graphs,” Comput. Biol. ‎Med., 169: 107882, 2024.‎

[21] M. Nourahmadi, A. Rahimi, H. Sadeqi, “Designing a ‎stock recommender system using the collaborative filtering ‎algorithm for the Tehran stock exchange,” Financ. Res. ‎J., 26(2): 302-330, 2024‎.

[22] B. Yang, T. Liang, J. Xiong, C. Zhong, “Deep reinforcement learning based on transformer and U-Net framework for stock trading,” Knowl. Based Syst., 262: 110211, 2023.

[23] J. Zou, J. Lou, B. Wang, S. Liu, “A novel deep reinforcement learning based automated stock trading system using cascaded lstm networks,” Expert Syst. Appl., 242: 122801, 2024.

[24] F. F. He, C. T. Chen, S. H. Huang, “A multi-agent virtual market model for generalization in reinforcement learning based trading strategies,” Appl. Soft Comput., 134, 109985, 2023.

[25] S. Singh, V. Goyal, S. Goel, H. C. Taneja, “Deep reinforcement learning models for automated stock trading," in Advanced Production and Industrial Engineering, 27: 175, 2022.

[26] M. Taghian, A. Asadi, R. Safabakhsh, “Learning financial asset-specific trading rules via deep reinforcement learning,” Expert Syst. Appl., 195: 116523, 2022.

[27] L. K. Felizardo, F. C. L. Paiva, C. de Vita Graves, E. Y. Matsumoto, A. H. R. Costa, E. Del-Moral-Hernandez, P. Brandimarte, “Outperforming algorithmic trading reinforcement learning systems: A supervised approach to the cryptocurrency market,” Expert Syst. Appl., 202: 117259, 2022.

[28] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018.

[29] T. Faturohman, T. Nugraha, “Islamic stock portfolio optimization using deep reinforcement learning,” J. Islamic Monetary Econ. Finance, 8(2): 181-200, 2022.

[30] H. Yue, J. Liu, D. Tian, Q. Zhang, “A novel anti-risk method for portfolio trading using deep reinforcement learning,” Electronics, 11(9): 1506, 2022.

[31] A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. De Maria et al., “Massively parallel methods for deep reinforcement learning,” arXiv preprint arXiv:1507.04296, 2015.

[32] Y. Shoham, K. Leyton-Brown, “Multiagent systems: Algorithmic, game-theoretic, and logical foundations,” Cambridge University Press, 2008.

[33] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, I. Mordatch, “Multi-agent actor-critic for mixed cooperative-ompetitive environments,” Adv. Neural Inf. Processing Syst., 30, 2017.

[34] S. Khonsha, M. A. Sarram, R. Sheikhpour, “A profitable portfolio allocation strategy based on money net-flow adjusted deep reinforcement learning,” Iran. J. Finance, 7(4): 59-89, 2023.

[35] H. Hu, L. Tang, S. Zhang, H. Wang, “Predicting the ‎direction of stock markets using optimized neural networks with ‎Google Trends,” Neurocomput., 285: 188-195, 2018.

[36] J. Schulman, F. ‎Wolski, P. Dhariwal, A. Radford, O. Klimov, “‎Proximal policy optimization algorithms,” arXiv preprint ‎arXiv:1707.06347, 2018.‎

LETTERS TO EDITOR

Journal of Electrical and Computer Engineering Innovations (JECEI) welcomes letters to the editor for the post-publication discussions and corrections which allows debate post publication on its site, through the Letters to Editor. Letters pertaining to manuscript published in JECEI should be sent to the editorial office of JECEI within three months of either online publication or before printed publication, except for critiques of original research. Following points are to be considering before sending the letters (comments) to the editor.

[1] Letters that include statements of statistics, facts, research, or theories should include appropriate references, although more than three are discouraged.

[2] Letters that are personal attacks on an author rather than thoughtful criticism of the author’s ideas will not be considered for publication.

[3] Letters can be no more than 300 words in length.

[4] Letter writers should include a statement at the beginning of the letter stating that it is being submitted either for publication or not.

[5] Anonymous letters will not be considered.

[6] Letter writers must include their city and state of residence or work.

[7] Letters will be edited for clarity and length.

Name *

Email Address *

Affiliation *

Comments *

Security Code *

Journal of Electrical and Computer Engineering Innovations (JECEI)

A Robust Concurrent Multi-Agent Deep Reinforcement Learning ‎based Stock Recommender System

References

References

Send comment about this article

Volume 13, Issue 1
January 2025
Pages 225-240

A Robust Concurrent Multi-Agent Deep Reinforcement Learning ‎based Stock Recommender System

References

References

Send comment about this article

Volume 13, Issue 1January 2025Pages 225-240

Volume 13, Issue 1
January 2025
Pages 225-240