AIBotTrade: A Comparative Empirical Study of Machine Learning, LSTM, and Proximal Policy Optimization for Automated Trading on NSE-Proxy Equity Data

Rahul Dhyani; Ashima Narang; Shelly Garg

doi:10.55524/ijirem.2026.13.3.3

Abstract

Most research papers on trading with ai gives good accuracy in good market on their test results but they don’t include the transaction cost so their results look better than reality. To tackle this problem, I build an AIBotTrade and tested three algorithms GBM, LSTM, PPO reinforcement learning on same data which makes profit on other studies which also include transaction cost of 10 basis point on every trade. I have used NSE India data from 2018-2022. I have Chosen this data on purpose because this time period includes COVID-19 crash, recovery after covid and from July 2022 to December 2023 a bear market and sideways market in between which other paper completely avoid. Now after the training the results were surprising to me LSTM gave best results with 42.30% return with Sharpe ratio 0.8301 by making only 63 trades. then GBM also made profit with return of 27% despite having only 35% directional accuracy. But somehow PPO reinforcement learning perform worst it gave negative Sharpe ratio because it will keep buying stocks in falling market maybe it calculates that stock will always rebound. So, the main lesson from this is in stock market trading prediction accuracy does no matter much I mean they do but in my findings transaction costs and market regime gives another picture entirely so these small 10 basis transaction costs and market regime can draw down the results of profitable strategy into loss one.

Keywords

Algorithmic trading, Equity market prediction, LSTM Long short-term memory networks, NSE India, PPO reinforcement learning, Regime mismatch, Sharpe ratio

References

S. Ayala-Hernandez, J. M. de la Fuente, M. C. Sánchez, and others, “Synergizing quantitative finance models and market microstructure analysis for enhanced algorithmic trading,” Journal of Open Innovation: Technology, Market, and Complexity, vol. 10, no. 4, Art. no. 100334, 2024. Available from: https://doi.org/10.1016/j.joitmc.2024.100334
Securities and Exchange Board of India (SEBI), Study on Analysis of Profit and Loss of Individual Traders Dealing in Equity F&O Segment. Mumbai, India, Jan. 2023. Available from: https://www.sebi.gov.in/reports-and-statistics/research/jan-2023/study-analysis-of-profit-and-loss-of-individual-traders-dealing-in-equity-f-o-segment_67525.html
M. L. de Prado, Advances in Financial Machine Learning. Hoboken, NJ, USA: John Wiley & Sons, 2018. Available from: https://tinyurl.com/37ma7dxd
W. Huang, Y. Nakamori, and S.-Y. Wang, “Forecasting stock market movement direction with support vector machine,” Computers & Operations Research, vol. 32, no. 10, pp. 2513–2522, Oct. 2005. Available from: https://doi.org/10.1016/j.cor.2004.03.016
J. Patel, S. Shah, P. Thakkar, and K. Kotecha, “Predicting stock and stock price index movement using trend deterministic data preparation and machine learning techniques,” Expert Systems with Applications, vol. 42, no. 1, pp. 259–268, Jan. 2015. Available from: https://www.sciencedirect.com/science/article/abs/pii/S0957417414006551
T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), San Francisco, CA, USA, 2016, pp. 785–794. Available from: https://dl.acm.org/doi/10.1145/2939672.2939785
V. Bajaj, A. Sharma, R. Singh, and others, “Exploring predictive prowess of ensemble machine learning models in banking stocks,” IIMB Management Review, May 2025. Available from: https://www.sciencedirect.com/journal/iimb-management-review
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997. Available from: https://ieeexplore.ieee.org/abstract/document/6795963
T. Fischer and C. Krauss, “Deep learning with long short-term memory networks for financial market predictions,” European Journal of Operational Research, vol. 270, no. 2, pp. 654–669, Oct. 2018. Available from: https://www.sciencedirect.com/science/article/abs/pii/S0377221717310652
S. K. Hiba Sadia, M. R. Hossain, M. A. Rahman, and others, “Stock Market Price Prediction: A Hybrid LSTM and Sequential Self-Attention Based Approach,” 2023. Available from: https://www.taylorfrancis.com/chapters/edit/10.1201/9781003467199-11/stock-market-price-prediction-karan-pardeshi-sukhpal-singh-gill-ahmed-abdelmoniem
Lim, S. Arik, N. Loeff, and T. Pfister, “Temporal Fusion Transformers for interpretable multi-horizon time series forecasting,” International Journal of Forecasting, vol. 37, no. 4, pp. 1748–1764, Oct.–Dec. 2021. Available from: https://www.sciencedirect.com/science/article/pii/S0169207021000637
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA, USA: MIT Press, 2018. Available from: https://doi.org/10.1017/S0263574799271172
T. Theate and D. Ernst, “An application of deep reinforcement learning to algorithmic trading,” Expert Systems with Applications, vol. 173, Art. no. 114632, 2021. Available from: https://www.sciencedirect.com/science/article/abs/pii/S0957417421000737
Author withheld for review, “Reinforcement Learning in Financial Decision Making: A Systematic Review,” arXiv preprint arXiv:2512.10913, Dec. 2025. Available from: https://arxiv.org/abs/2512.10913
K. Kirtac and G. Germano, “Sentiment trading with large language models,” Finance Research Letters, vol. 69, Art. no. 105227, 2024. Available from: https://doi.org/10.1016/j.frl.2024.105227
W. Lo, “The adaptive markets hypothesis: Market efficiency from an evolutionary perspective,” Journal of Portfolio Management, vol. 30, no. 5, pp. 15–29, 2004. Available from: http://stat.wharton.upenn.edu/~steele/Courses/434/434Context/EfficientMarket/AndyLoJPM2004.pdf
Securities and Exchange Board of India (SEBI), “Framework for Algorithmic Trading,” Circular CIR/MRD/DP/09/2013, Mar. 2013. Available from: https://www.sebi.gov.in/legal/circulars/mar-2013/broad-guidelines-on-algorithmic-trading_24407.html
G. Konstantinidis, M. Vassiliou, A. Karathanasis, and others, “FinLlama: Financial sentiment analysis for algorithmic trading,” in Proc. ACM Int. Conf. AI in Finance (ICAIF), Nov. 2024. Available from: https://arxiv.org/abs/2403.12285
X. Liu, H. Zhang, Y. Wang, and others, “FinGPT: Open-source financial large language models,” arXiv preprint arXiv:2306.06031, 2023. Available from: https://arxiv.org/abs/2306.06031
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv preprint arXiv:1707.06347, 2017. Available from: https://arxiv.org/abs/1707.06347
H. Sagong, H. Kim, and H. Hong, “Hi-DARTS: Hierarchical Dynamically Adapting Reinforcement Trading System,” arXiv preprint arXiv:2509.12048, 2025. Available from: https://arxiv.org/abs/2509.12048
Z. Liu, Y. Chen, J. Wang, and others, “Comparative analysis of LSTM, GRU, and Transformer models for stock price prediction,” 2024. Available from: https://dl.acm.org/doi/full/10.1145/3700058.3700075

Cites this article as

R. Dhyani, A. Narang, S. Garg, "AIBotTrade: A Comparative Empirical Study of Machine Learning, LSTM, and Proximal Policy Optimization for Automated Trading on NSE-Proxy Equity Data ", International Journal of Innovative Research in Engineering and Management (IJIREM), Vol-13, Issue-3, Page No-19-27, 2026. Available from: https://doi.org/10.55524/ijirem.2026.13.3.3