A gray-box DPDA-based intrusion detection technique using system-call monitoring

In this paper, we present a novel technique for automatic and efficient intrusion detection based on learning program behaviors. Program behavior is captured in terms of issued system calls augmented with point-of-system-call information, and is modeled according to an efficient deterministic pushdown automaton (DPDA). The frequency of visit of each state is captured and statistically analyzed to detect abnormal execution patterns. This approach provides a very accurate learning of program behavior, which avoids a broad class of impossible path exploits. It also allows detection of new classes of attacks such as denial-of-service and brute-force dictionary attacks. We also present a complexity analysis of our model, and show that its time and space complexity is polynomial and fairly comparable to other similar approaches in learning, and hugely better in detection. Moreover, We evaluate our approach experimentally in terms of false positive rate, convergence rate, and performance. Finally, We shall discuss classes of attacks which are detectable and undetectable by our approach.