Flow-based Compromise Detection

R.J. Hofstede

Research output: ThesisPhD Thesis - Research UT, graduation UT

Abstract

Brute-force attacks are omnipresent and manyfold on the Internet, and aim at compromising user accounts by issuing large numbers of authentication attempts on applications and daemons. Widespread targets of such attacks are Secure SHell (SSH) and Web applications, for example. The impact of brute-force attacks and compromises resulting thereof is often severe: Once compromised, attackers gain access to remote machines, allowing those machines to be misused for all sorts of criminal activities, such as sharing illegal content and participating in Distributed Denial of Service (DDoS) attacks. While the number of brute-force attacks is ever-increasing, we have seen that only few brute-force attacks actually result in a compromise. Those compromised devices are however those that require attention by security teams, as they may be misused for all sorts of malicious activities. We therefore propose a new paradigm in this thesis for monitoring network security incidents: compromise detection. Compromise detection allows security teams to focus on what is really important, namely detecting those hosts that have been compromised instead of all hosts that have been attacked. Speaking metaphorically, one could say that we target scored goals, instead of just shots on goals. A straightforward approach for compromise detection would be host-based, by analyzing network traffic and log files on individual hosts. Although this typically yields high detection accuracies, it is infeasible in large networks; These networks may comprise thousands of hosts, controlled by many persons, on which agents need to be installed. In addition, host-based approaches lack a global attack view, i.e., which hosts in the same network have been contacted by the same attacker. We therefore take a network-based approach, where sensors are deployed at strategic observation points in the network. The traditional approach would be packet-based, but both high link speeds and high data rates make the deployment of packet-based approaches rather expensive. In addition, the fact that more and more traffic is encrypted renders the analysis of full packets useless. Flow-based approaches, however, aggregate individual packets into flows, providing major advantages in terms of scalability and deployment. The main contribution of this thesis is to prove that flow-based compromise detection is viable. Our approach consists of several steps. First, we select two target applications, Web applications and SSH, which we found to be important targets of attacks on the Internet because of the high impact of a compromise and their wide deployment. Second, we analyze protocol behavior, attack tools and attack traffic to better understand the nature of these attacks. Third, we develop software for validating our algorithms and approach. Besides using this software for our own validations (i.e., in which we use log files as ground-truth), our open-source Intrusion Detection System (IDS) SSHCure is extensively used by other parties, allowing us to validate our approach on a much broader basis. Our evaluations, performed on Internet traffic, have shown that we can achieve detection accuracies between 84% and 100%, depending on the protocol used by the target application, quality of the dataset, and the type of the monitored network. Also, the wide deployment of SSHCure, as well as other prototype deployments in real networks, have shown that our algorithms can actually be used in production deployments. As such, we conclude that flow-based compromise detection is viable on the Internet.
LanguageUndefined
Awarding Institution
  • University of Twente
Supervisors/Advisors
  • Pras, Aiko , Supervisor
  • Dreo Rodosek, G., Supervisor
  • Rodosek, G. Dreo, Advisor
Award date29 Jun 2016
Place of PublicationEnschede
Publisher
Print ISBNs978-90-365-4066-7
DOIs
StatePublished - 29 Jun 2016

Keywords

  • EWI-27037
  • METIS-317117
  • EC Grant Agreement nr.: FP7/257513
  • EC Grant Agreement nr.: FP7/318488
  • EC Grant Agreement nr.: FP/313296
  • IR-100543

Cite this

Hofstede, R. J. (2016). Flow-based Compromise Detection Enschede: Centre for Telematics and Information Technology (CTIT) DOI: 10.3990/1.9789036540667
Hofstede, R.J.. / Flow-based Compromise Detection. Enschede : Centre for Telematics and Information Technology (CTIT), 2016. 198 p.
@phdthesis{790e5c6a519e49a0a41b59107c9f2efd,
title = "Flow-based Compromise Detection",
abstract = "Brute-force attacks are omnipresent and manyfold on the Internet, and aim at compromising user accounts by issuing large numbers of authentication attempts on applications and daemons. Widespread targets of such attacks are Secure SHell (SSH) and Web applications, for example. The impact of brute-force attacks and compromises resulting thereof is often severe: Once compromised, attackers gain access to remote machines, allowing those machines to be misused for all sorts of criminal activities, such as sharing illegal content and participating in Distributed Denial of Service (DDoS) attacks. While the number of brute-force attacks is ever-increasing, we have seen that only few brute-force attacks actually result in a compromise. Those compromised devices are however those that require attention by security teams, as they may be misused for all sorts of malicious activities. We therefore propose a new paradigm in this thesis for monitoring network security incidents: compromise detection. Compromise detection allows security teams to focus on what is really important, namely detecting those hosts that have been compromised instead of all hosts that have been attacked. Speaking metaphorically, one could say that we target scored goals, instead of just shots on goals. A straightforward approach for compromise detection would be host-based, by analyzing network traffic and log files on individual hosts. Although this typically yields high detection accuracies, it is infeasible in large networks; These networks may comprise thousands of hosts, controlled by many persons, on which agents need to be installed. In addition, host-based approaches lack a global attack view, i.e., which hosts in the same network have been contacted by the same attacker. We therefore take a network-based approach, where sensors are deployed at strategic observation points in the network. The traditional approach would be packet-based, but both high link speeds and high data rates make the deployment of packet-based approaches rather expensive. In addition, the fact that more and more traffic is encrypted renders the analysis of full packets useless. Flow-based approaches, however, aggregate individual packets into flows, providing major advantages in terms of scalability and deployment. The main contribution of this thesis is to prove that flow-based compromise detection is viable. Our approach consists of several steps. First, we select two target applications, Web applications and SSH, which we found to be important targets of attacks on the Internet because of the high impact of a compromise and their wide deployment. Second, we analyze protocol behavior, attack tools and attack traffic to better understand the nature of these attacks. Third, we develop software for validating our algorithms and approach. Besides using this software for our own validations (i.e., in which we use log files as ground-truth), our open-source Intrusion Detection System (IDS) SSHCure is extensively used by other parties, allowing us to validate our approach on a much broader basis. Our evaluations, performed on Internet traffic, have shown that we can achieve detection accuracies between 84{\%} and 100{\%}, depending on the protocol used by the target application, quality of the dataset, and the type of the monitored network. Also, the wide deployment of SSHCure, as well as other prototype deployments in real networks, have shown that our algorithms can actually be used in production deployments. As such, we conclude that flow-based compromise detection is viable on the Internet.",
keywords = "EWI-27037, METIS-317117, EC Grant Agreement nr.: FP7/257513, EC Grant Agreement nr.: FP7/318488, EC Grant Agreement nr.: FP/313296, IR-100543",
author = "R.J. Hofstede",
year = "2016",
month = "6",
day = "29",
doi = "10.3990/1.9789036540667",
language = "Undefined",
isbn = "978-90-365-4066-7",
publisher = "Centre for Telematics and Information Technology (CTIT)",
address = "Netherlands",
school = "University of Twente",

}

Hofstede, RJ 2016, 'Flow-based Compromise Detection', University of Twente, Enschede. DOI: 10.3990/1.9789036540667

Flow-based Compromise Detection. / Hofstede, R.J.

Enschede : Centre for Telematics and Information Technology (CTIT), 2016. 198 p.

Research output: ThesisPhD Thesis - Research UT, graduation UT

TY - THES

T1 - Flow-based Compromise Detection

AU - Hofstede,R.J.

PY - 2016/6/29

Y1 - 2016/6/29

N2 - Brute-force attacks are omnipresent and manyfold on the Internet, and aim at compromising user accounts by issuing large numbers of authentication attempts on applications and daemons. Widespread targets of such attacks are Secure SHell (SSH) and Web applications, for example. The impact of brute-force attacks and compromises resulting thereof is often severe: Once compromised, attackers gain access to remote machines, allowing those machines to be misused for all sorts of criminal activities, such as sharing illegal content and participating in Distributed Denial of Service (DDoS) attacks. While the number of brute-force attacks is ever-increasing, we have seen that only few brute-force attacks actually result in a compromise. Those compromised devices are however those that require attention by security teams, as they may be misused for all sorts of malicious activities. We therefore propose a new paradigm in this thesis for monitoring network security incidents: compromise detection. Compromise detection allows security teams to focus on what is really important, namely detecting those hosts that have been compromised instead of all hosts that have been attacked. Speaking metaphorically, one could say that we target scored goals, instead of just shots on goals. A straightforward approach for compromise detection would be host-based, by analyzing network traffic and log files on individual hosts. Although this typically yields high detection accuracies, it is infeasible in large networks; These networks may comprise thousands of hosts, controlled by many persons, on which agents need to be installed. In addition, host-based approaches lack a global attack view, i.e., which hosts in the same network have been contacted by the same attacker. We therefore take a network-based approach, where sensors are deployed at strategic observation points in the network. The traditional approach would be packet-based, but both high link speeds and high data rates make the deployment of packet-based approaches rather expensive. In addition, the fact that more and more traffic is encrypted renders the analysis of full packets useless. Flow-based approaches, however, aggregate individual packets into flows, providing major advantages in terms of scalability and deployment. The main contribution of this thesis is to prove that flow-based compromise detection is viable. Our approach consists of several steps. First, we select two target applications, Web applications and SSH, which we found to be important targets of attacks on the Internet because of the high impact of a compromise and their wide deployment. Second, we analyze protocol behavior, attack tools and attack traffic to better understand the nature of these attacks. Third, we develop software for validating our algorithms and approach. Besides using this software for our own validations (i.e., in which we use log files as ground-truth), our open-source Intrusion Detection System (IDS) SSHCure is extensively used by other parties, allowing us to validate our approach on a much broader basis. Our evaluations, performed on Internet traffic, have shown that we can achieve detection accuracies between 84% and 100%, depending on the protocol used by the target application, quality of the dataset, and the type of the monitored network. Also, the wide deployment of SSHCure, as well as other prototype deployments in real networks, have shown that our algorithms can actually be used in production deployments. As such, we conclude that flow-based compromise detection is viable on the Internet.

AB - Brute-force attacks are omnipresent and manyfold on the Internet, and aim at compromising user accounts by issuing large numbers of authentication attempts on applications and daemons. Widespread targets of such attacks are Secure SHell (SSH) and Web applications, for example. The impact of brute-force attacks and compromises resulting thereof is often severe: Once compromised, attackers gain access to remote machines, allowing those machines to be misused for all sorts of criminal activities, such as sharing illegal content and participating in Distributed Denial of Service (DDoS) attacks. While the number of brute-force attacks is ever-increasing, we have seen that only few brute-force attacks actually result in a compromise. Those compromised devices are however those that require attention by security teams, as they may be misused for all sorts of malicious activities. We therefore propose a new paradigm in this thesis for monitoring network security incidents: compromise detection. Compromise detection allows security teams to focus on what is really important, namely detecting those hosts that have been compromised instead of all hosts that have been attacked. Speaking metaphorically, one could say that we target scored goals, instead of just shots on goals. A straightforward approach for compromise detection would be host-based, by analyzing network traffic and log files on individual hosts. Although this typically yields high detection accuracies, it is infeasible in large networks; These networks may comprise thousands of hosts, controlled by many persons, on which agents need to be installed. In addition, host-based approaches lack a global attack view, i.e., which hosts in the same network have been contacted by the same attacker. We therefore take a network-based approach, where sensors are deployed at strategic observation points in the network. The traditional approach would be packet-based, but both high link speeds and high data rates make the deployment of packet-based approaches rather expensive. In addition, the fact that more and more traffic is encrypted renders the analysis of full packets useless. Flow-based approaches, however, aggregate individual packets into flows, providing major advantages in terms of scalability and deployment. The main contribution of this thesis is to prove that flow-based compromise detection is viable. Our approach consists of several steps. First, we select two target applications, Web applications and SSH, which we found to be important targets of attacks on the Internet because of the high impact of a compromise and their wide deployment. Second, we analyze protocol behavior, attack tools and attack traffic to better understand the nature of these attacks. Third, we develop software for validating our algorithms and approach. Besides using this software for our own validations (i.e., in which we use log files as ground-truth), our open-source Intrusion Detection System (IDS) SSHCure is extensively used by other parties, allowing us to validate our approach on a much broader basis. Our evaluations, performed on Internet traffic, have shown that we can achieve detection accuracies between 84% and 100%, depending on the protocol used by the target application, quality of the dataset, and the type of the monitored network. Also, the wide deployment of SSHCure, as well as other prototype deployments in real networks, have shown that our algorithms can actually be used in production deployments. As such, we conclude that flow-based compromise detection is viable on the Internet.

KW - EWI-27037

KW - METIS-317117

KW - EC Grant Agreement nr.: FP7/257513

KW - EC Grant Agreement nr.: FP7/318488

KW - EC Grant Agreement nr.: FP/313296

KW - IR-100543

U2 - 10.3990/1.9789036540667

DO - 10.3990/1.9789036540667

M3 - PhD Thesis - Research UT, graduation UT

SN - 978-90-365-4066-7

PB - Centre for Telematics and Information Technology (CTIT)

CY - Enschede

ER -

Hofstede RJ. Flow-based Compromise Detection. Enschede: Centre for Telematics and Information Technology (CTIT), 2016. 198 p. Available from, DOI: 10.3990/1.9789036540667