Intelligent Multi-Agent Deep Reinforcement Learning for Autonomous Microgrid Management: Enhancing Energy Resilience through Solar-Wind-Battery Integration in Urban Networks

Adel Elgammal

doi:10.55524/ijirem.2026.13.3.13

Abstract

Current urban power systems are facing dual challenges amid the process of integrating renewable energy: continuous growth in energy demand, and compromised grid stability and energy security under the impact of climate disasters. Traditional centralized control systems cannot adapt to the complex dynamic interactions of distributed photovoltaics, wind turbines, and energy storage units within urban microgrids, so energy resilience vulnerabilities are easily exposed during peak load periods and extreme weather scenarios. To address this problem, this study proposes a new multi-agent deep reinforcement learning (MADRL) framework that solves the core flaws of traditional systems through decentralized decision-making and adaptive control mechanisms. This framework uses a hybrid Actor-Critic architecture integrated with graph neural networks, and relies on a multi-objective reward mechanism to simultaneously optimize three core indicators: energy cost, carbon emissions, and grid stability. This study conducts simulation verification on a typical urban block microgrid configured with 500kW photovoltaics, 150kW wind power, and 400kWh battery energy storage. Taking traditional rule-based control systems as the comparison baseline, Simulation results show that the proposed framework improves energy cost efficiency by 23%, reduces carbon emissions by 18%, and increases grid disturbance response speed by 35%. During peak production periods, the renewable energy utilization rate reaches 94%; when sudden load changes occur, grid frequency remains stable within ±0.2Hz. Furthermore, the framework can be deployed in a scalable way to adapt to the weather patterns, electricity consumption characteristics, and grid infrastructure constraints of different cities. This study has certain limitations: it requires high computing power to implement in real time, and it relies on large amounts of training data. This framework can be applied to urban energy planning, smart city construction, and climate change mitigation strategies, and can deliver social values including improved energy security, reduced user electricity bills, and enhanced environmental sustainability of urban communities.

Keywords

Microgrid Control, Renewable Energy Integration, Smart Grid Optimization, Energy Resilience Urban Decarbonization, Machine Learning for Energy Management

References

M. Shahidehpour, Z. Li, and F. Aminifar, “Hybrid AC/DC microgrids: A pathway to smart grid modernization,” IEEE Transactions on Smart Grid, vol. 14, no. 3, pp. 1789–1803, May 2023. Available from: https://doi.org/10.1109/TSG.2023.3251847
International Energy Agency, World Energy Outlook 2023: Urban Energy Transitions. Paris, France: IEA Publications, 2023, pp. 127–156. Available from: https://doi.org/10.1787/827374a7-en
Kumar, T. Saha, and N. Hatziargyriou, “Microgrid planning under uncertainty: A comprehensive review of methods and applications,” Renewable and Sustainable Energy Reviews, vol. 164, pp. 112467–112489, Sep. 2022. Available from: https://doi.org/10.1016/j.rser.2022.112467
S. Parhizi, H. Lotfi, A. Khodaei, and S. Bahramirad, “State of the art in research on microgrids: A review,” IEEE Access, vol. 3, pp. 890–925, Jul. 2015. Available from: https://doi.org/10.1109/ACCESS.2015.2443119
BloombergNEF, Global Microgrid Market Outlook 2023. New York, NY: Bloomberg Finance L.P., 2023, pp. 23–45.
T. Hirsch, A. Parag, and J. Guerrero, “Microgrids: A review of technologies, key drivers, and outstanding issues,” Renewable and Sustainable Energy Reviews, vol. 90, pp. 402–411, Jul. 2018. Available from: https://doi.org/10.1016/j.rser.2018.03.040
International Renewable Energy Agency (IRENA), Renewable Power Generation Costs in 2022. Abu Dhabi: IRENA, 2023, pp. 78–102. Available from: https://doi.org/10.1787/5e0de7c3-en
Viswanathan and M. Kintner-Meyer, “Second use of transportation batteries: Maximizing the value of batteries for transportation and grid services,” IEEE Transactions on Vehicular Technology, vol. 60, no. 7, pp. 2963–2970, Sep. 2011. Available from: https://doi.org/10.1109/TVT.2011.2160378
E. Planas, A. Gil-de-Muro, J. Andreu, I. Kortabarria, and I. Martínez de Alegría, “General aspects, hierarchical controls and droop methods in microgrids: A review,” Renewable and Sustainable Energy Reviews, vol. 17, pp. 147–159, Jan. 2013. Available from: https://doi.org/10.1016/j.rser.2012.09.032
J. M. Guerrero, J. C. Vasquez, J. Matas, L. G. de Vicuña, and M. Castilla, “Hierarchical control of droop-controlled AC and DC microgrids—A general approach toward standardization,” IEEE Transactions on Industrial Electronics, vol. 58, no. 1, pp. 158–172, Jan. 2011. Available from: https://doi.org/10.1109/TIE.2010.2066534
F. Katiraei, R. Iravani, N. Hatziargyriou, and A. Dimeas, “Microgrids management,” IEEE Power & Energy Magazine, vol. 6, no. 3, pp. 54–65, May–Jun. 2008. Available from: https://doi.org/10.1109/MPE.2008.918702
Y. Zhang, T. Huang, and E. F. Bompard, “Big data analytics in smart grids: A review,” Energy Informatics, vol. 1, no. 1, pp. 8–24, Dec. 2018. Available from: https://doi.org/10.1186/s42162-018-0013-2
H. Farzin, M. Fotuhi-Firuzabad, and M. Moeini-Aghtaie, “A practical scheme to involve degradation cost of lithium-ion batteries in vehicle-to-grid applications,” IEEE Transactions on Sustainable Energy, vol. 7, no. 4, pp. 1730–1738, Oct. 2016. Available from: https://doi.org/10.1109/TSTE.2016.2558500
S. M. Dawoud, X. Lin, and M. I. Okba, “Hybrid renewable microgrid optimization techniques: A review,” Renewable and Sustainable Energy Reviews, vol. 82, pp. 2039–2052, Feb. 2018. Available from: https://doi.org/10.1016/j.rser.2017.08.007
M. Marzband, F. Azarinejadian, M. Savaghebi, and J. M. Guerrero, “An optimal energy management system for islanded microgrids based on multiperiod artificial bee colony combined with Markov chain,” IEEE Systems Journal, vol. 11, no. 3, pp. 1712–1722, Sep. 2017. Available from: https://doi.org/10.1109/JSYST.2015.2422253
Parisio, E. Rikos, and L. Glielmo, “A model predictive control approach to microgrid operation optimization,” IEEE Transactions on Control Systems Technology, vol. 22, no. 5, pp. 1813–1827, Sep. 2014. Available from: https://doi.org/10.1109/TCST.2013.2295737
E. Dall’Anese, S. V. Dhople, B. B. Johnson, and F. Dörfler, “Decentralized optimal dispatch of photovoltaic inverters in residential distribution systems,” IEEE Transactions on Energy Conversion, vol. 29, no. 4, pp. 957–967, Dec. 2014. Available from: https://doi.org/10.1109/TEC.2014.2357997
T. Logenthiran, D. Srinivasan, A. M. Khambadkone, and H. N. Aung, “Multiagent system for real-time operation of a microgrid in real-time digital simulator,” IEEE Transactions on Smart Grid, vol. 3, no. 2, pp. 925–933, Jun. 2012. Available from: https://doi.org/10.1109/TSG.2012.2189028
N. Hatziargyriou, H. Asano, R. Iravani, and C. Marnay, “Microgrids,” IEEE Power & Energy Magazine, vol. 5, no. 4, pp. 78–94, Jul.–Aug. 2007. Available from: https://doi.org/10.1109/MPAE.2007.376583
N. Papadimitriou, E. I. Zountouridou, and N. D. Hatziargyriou, “Review of hierarchical control in DC microgrids,” Electric Power Systems Research, vol. 122, pp. 159–167, May 2015. Available from: https://doi.org/10.1016/j.epsr.2015.01.006
Q. Shafiee, J. M. Guerrero, and J. C. Vasquez, “Distributed secondary control for islanded microgrids—A novel approach,” IEEE Transactions on Power Electronics, vol. 29, no. 2, pp. 1018–1031, Feb. 2014. Available from: https://doi.org/10.1109/TPEL.2013.2259506
F. Gao and M. R. Iravani, “A control strategy for a distributed generation unit in grid-connected and autonomous modes of operation,” IEEE Transactions on Power Delivery, vol. 23, no. 2, pp. 850–859, Apr. 2008. Available from: https://doi.org/10.1109/TPWRD.2007.915950
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA: MIT Press, 2018, pp. 142–189. Available from: https://mitpress.mit.edu/9780262039246/reinforcement-learning-an-introduction/
Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015. Available from: https://doi.org/10.1038/nature14236
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, Jul. 2017. Available from: https://arxiv.org/abs/1707.06347
T. P. Lillicrap et al., “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, Sep. 2015. Available from: https://arxiv.org/abs/1509.02971
G. Tampuu et al., “Multiagent deep reinforcement learning with extremely sparse rewards,” arXiv preprint arXiv:1707.01068, Jul. 2017. Available from: https://arxiv.org/abs/1707.01068
K. Zhang, Z. Yang, and T. Ba?ar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” in Handbook of Reinforcement Learning and Control, K. G. Vamvoudakis, Y. Wan, F. L. Lewis, and D. Cansever, Eds. Cham, Switzerland: Springer, 2021, pp. 321–384. Available from: https://doi.org/10.1007/978-3-030-60990-0_12
R. Lowe, Y. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 6379–6390. Available from: https://proceedings.neurips.cc/paper/2017
J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in Proceedings of the 32nd AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, Feb. 2018, pp. 2974–2982. Available from: https://doi.org/10.1609/aaai.v32i1.11794
P. Sunehag et al., “Value-decomposition networks for cooperative multi-agent learning,” arXiv preprint arXiv:1706.05296, Jun. 2017. Available from: https://arxiv.org/abs/1706.05296
T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, and S. Whiteson, “QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, Jul. 2018, pp. 4295–4304. Available from: https://proceedings.mlr.press/v80/rashid18a.html
J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang, “QPLEX: Duplex dueling multi-agent Q-learning,” arXiv preprint arXiv:2008.01062, Aug. 2020. Available from: https://arxiv.org/abs/2008.01062
S. Omidshafiei et al., “Deep multi-agent reinforcement learning for partially observable parameterized environments,” arXiv preprint arXiv:1705.06404, May 2017. Available from: https://arxiv.org/abs/1705.06404
C. Yu, A. Velu, E. Vinitsky, Y. Gao, L. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of PPO in cooperative multi-agent games,” arXiv preprint arXiv:2103.01955, Mar. 2021. Available from: https://arxiv.org/abs/2103.01955
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, Dec. 2014. Available from: https://arxiv.org/abs/1412.6980
V. Mnih et al., “Asynchronous methods for deep reinforcement learning,” in Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, Jun. 2016, pp. 1928–1937. Available from: https://proceedings.mlr.press/v48/mniha16.html
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in Proceedings of the 32nd International Conference on Machine Learning, Lille, France, Jul. 2015, pp. 1889–1897. Available from: https://proceedings.mlr.press/v37/schulman15.html
S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, Jul. 2018, pp. 1587–1596. Available from: https://proceedings.mlr.press/v80/fujimoto18a.html
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, Jul. 2018, pp. 1861–1870. Available from: https://proceedings.mlr.press/v80/haarnoja18b.html
M. Hessel et al., “Rainbow: Combining improvements in deep reinforcement learning,” in Proceedings of the 32nd AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, Feb. 2018, pp. 3215–3222. Available from: https://doi.org/10.1609/aaai.v32i1.11796
Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, “Dueling network architectures for deep reinforcement learning,” in Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, Jun. 2016, pp. 1995–2003. Available from: https://proceedings.mlr.press/v48/wangf16.html
T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” arXiv preprint arXiv:1511.05952, Nov. 2015. Available from: https://arxiv.org/abs/1511.05952
H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA, Feb. 2016, pp. 2094–2100. Available from: https://doi.org/10.1609/aaai.v30i1.10295
M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” in Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, Aug. 2017, pp. 449–458. Available from: https://proceedings.mlr.press/v70/bellemare17a.html
G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos, “Count-based exploration with neural density models,” in Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, Aug. 2017, pp. 2721–2730. Available from: https://proceedings.mlr.press/v70/ostrovski17a.html
Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems, vol. 32, 2019, pp. 8026–8037. Available from: https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
M. Abadi et al., “TensorFlow: A system for large-scale machine learning,” in Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, Savannah, GA, USA, Nov. 2016, pp. 265–283. Available from: https://www.usenix.org/conference/osdi16/technical-sessions/presentation/abadi
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, May 2015. Available from: https://doi.org/10.1038/nature14539
Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA: MIT Press, 2016, pp. 326–380. Available from: https://www.deeplearningbook.org
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, May 2010, pp. 249–256. Available from: https://proceedings.mlr.press/v9/glorot10a.html
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proceedings of the 32nd International Conference on Machine Learning, Lille, France, Jul. 2015, pp. 448–456. Available from: https://proceedings.mlr.press/v37/ioffe15.html
N. Srivastava et al., “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014. Available from: https://jmlr.org/papers/v15/srivastava14a.html
Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, vol. 25, 2012, pp. 1097–1105. Available from: https://proceedings.neurips.cc/paper/2012
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, Sep. 2014. Available from: https://arxiv.org/abs/1409.1556
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, Jun. 2016, pp. 770–778. Available from: https://doi.org/10.1109/CVPR.2016.90
Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 5998–6008. Available from: https://proceedings.neurips.cc/paper/2017
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, Oct. 2018. Available from: https://arxiv.org/abs/1810.04805
Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI Blog, Feb. 2019. Available from: https://openai.com/blog/better-language-models/
T. Brown et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, May 2020. Available from: https://arxiv.org/abs/2005.14165
X. Liu, A. Shahidehpour, and Z. Li, “Microcontroller-based microgrid energy management systems: Design principles and deployment challenges,” IEEE Transactions on Smart Grid, vol. 12, no. 3, pp. 2156–2168, May 2021. Available from: https://doi.org/10.1109/TSG.2020.3039876
P. Siano, “Demand response and smart grids—A survey,” Renewable and Sustainable Energy Reviews, vol. 30, pp. 461–478, Jun. 2014. Available from: https://doi.org/10.1016/j.rser.2013.10.022
S. Maharjan, Q. Zhu, Y. Zhang, S. Gjessing, and T. Ba?ar, “Dependence of demand response and renewable energy penetration on the electricity market,” IEEE Journal of Selected Topics in Signal Processing, vol. 8, no. 6, pp. 1031–1041, Dec. 2014. Available from: https://doi.org/10.1109/JSTSP.2014.2332296
D. E. Olivares et al., “Trends in microgrid control,” IEEE Transactions on Smart Grid, vol. 5, no. 4, pp. 1905–1919, Jul. 2014. Available from: https://doi.org/10.1109/TSG.2013.2295514
Oudalov, D. Chartouni, and C. Ohler, “Deploying active distribution networks for the smooth integration of distributed generation,” Energy Policy, vol. 35, no. 10, pp. 4999–5012, Oct. 2007. Available from: https://doi.org/10.1016/j.enpol.2007.04.003
R. H. Lasseter, “Smart distribution: Coupled microgrids,” Proceedings of the IEEE, vol. 99, no. 6, pp. 1074–1082, Jun. 2011. Available from: https://doi.org/10.1109/JPROC.2011.2106670
F. A. Cleveland, “Cyber security issues for control systems,” in IEEE Conference on Technologies for Homeland Security and Information Networking, Oct. 2011, pp. 1–6. Available from: https://doi.org/10.1109/CNSIV.2011.6080957

Cites this article as

A. Elgammal "Intelligent Multi-Agent Deep Reinforcement Learning for Autonomous Microgrid Management: Enhancing Energy Resilience through Solar-Wind-Battery Integration in Urban Networks", International Journal of Innovative Research in Engineering and Management (IJIREM), Vol-13, Issue-3, Page No-92-118, 2026. Available from: https://doi.org/10.55524/ijirem.2026.13.3.13