A Hybrid Deep Reinforcement and Machine Learning-Based Intrusion Detection System for Dynamic XSS Attacks

Abstract

Web-based systems are vulnerable to continuously evolving or self-updating attacks such as Cross-Site Scripting (XSS). Traditional Intrusion Detection Systems (IDS) provide limited protection against this threat through signature-based and anomaly-based methods. In this study, Machine Learning (ML) methods are used in conjunction with Deep Reinforcement Learning (DRL) techniques. In the proposed approach, ML methods are utilized to rapidly detect known attacks, while DRL provides adaptive learning against more general and unknown threats. These two components are trained independently and then make decisions through a weighted combination during the prediction phase. The aim is to address the shortcomings of current IDS systems in defending against dynamic XSS attacks. Experimental results show that, in real-time IDS environments, combining Random Forest with Word2Vec ensures detection within 10 ms, maintains an F1 score of about 0.99, and keeps computational cost minimal. In contrast, for offline or SOC-based setups where longer training and adaptive learning are acceptable, the DDQN-Word2Vec combination proves most effective. Overall, the proposed hybrid system delivers scalable, real-time protection against dynamic and zero-day web threats.