ENTRADA: A high-performance network traffic data streaming warehouse

Maarten Wullink; Giovane C.M. Moura; Moritz Müller; Cristian Hesselman

doi:10.1109/NOMS.2016.7502925

ENTRADA: A high-performance network traffic data streaming warehouse

Maarten Wullink
, Giovane C.M. Moura
, Moritz Müller
, Cristian Hesselman

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

48 Citations (Scopus)

Abstract

We present ENTRADA, a high-performance data streaming warehouse that enables researchers and operators to analyze vast amounts of network traffic and measurement data within interactive response times (seconds to few minutes), even in a small computer cluster. ENTRADA delivers such performance by employing a optimized file format and a high-performance query engine, both open-source. ENTRADA has been operational for more than 1.5 years, having ingested more than 100 TB of pcap files from two.nl DNS authoritative servers. As we discuss, we use this data in projects that aim at further increasing the security and stability of the.nl zone. We present in this paper our design choices, experiences, and a performance evaluation of ENTRADA. Finally, we open-source ENTRADA, which can be used out-of-the-box by researchers, operators, and registries to deploy their own networking analysis clusters for DNS traffic, and can be easily extended to handle any other structured data.

Original language	English
Title of host publication	Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium
Editors	Sema Oktug Badonnel, Mehmet Ulema, Cicek Cavdar, Lisandro Zambenedetti Granville, Carlos Raniery P. dos Santos
Publisher	IEEE
Pages	913-918
Number of pages	6
ISBN (Electronic)	9781509002238
DOIs	https://doi.org/10.1109/NOMS.2016.7502925
Publication status	Published - 30 Jun 2016
Externally published	Yes
Event	2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016: Managing Everything toward a Secure, Smart, and Hyperconnected World - Suleyman Demirel Cultural Center, Istanbul, Turkey Duration: 25 Apr 2016 → 29 Apr 2016 Conference number: 15 https://noms2016.ieee-noms.org/ http://noms2016.ieee-noms.org/

Conference

Conference	2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016
Abbreviated title	NOMS 2016
Country/Territory	Turkey
City	Istanbul
Period	25/04/16 → 29/04/16
Internet address	https://noms2016.ieee-noms.org/ http://noms2016.ieee-noms.org/

Keywords

n/a OA procedure

Access to Document

10.1109/NOMS.2016.7502925

Cite this

Wullink, M., Moura, G. C. M., Müller, M., & Hesselman, C. (2016). ENTRADA: A high-performance network traffic data streaming warehouse. In S. O. Badonnel, M. Ulema, C. Cavdar, L. Z. Granville, & C. R. P. dos Santos (Eds.), Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium (pp. 913-918). Article 7502925 IEEE. https://doi.org/10.1109/NOMS.2016.7502925

Wullink, Maarten ; Moura, Giovane C.M. ; Müller, Moritz et al. / ENTRADA : A high-performance network traffic data streaming warehouse. Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium. editor / Sema Oktug Badonnel ; Mehmet Ulema ; Cicek Cavdar ; Lisandro Zambenedetti Granville ; Carlos Raniery P. dos Santos. IEEE, 2016. pp. 913-918

@inproceedings{ab3934a832914025943f154ea3da011c,

title = "ENTRADA: A high-performance network traffic data streaming warehouse",

abstract = "We present ENTRADA, a high-performance data streaming warehouse that enables researchers and operators to analyze vast amounts of network traffic and measurement data within interactive response times (seconds to few minutes), even in a small computer cluster. ENTRADA delivers such performance by employing a optimized file format and a high-performance query engine, both open-source. ENTRADA has been operational for more than 1.5 years, having ingested more than 100 TB of pcap files from two.nl DNS authoritative servers. As we discuss, we use this data in projects that aim at further increasing the security and stability of the.nl zone. We present in this paper our design choices, experiences, and a performance evaluation of ENTRADA. Finally, we open-source ENTRADA, which can be used out-of-the-box by researchers, operators, and registries to deploy their own networking analysis clusters for DNS traffic, and can be easily extended to handle any other structured data.",

keywords = "n/a OA procedure",

author = "Maarten Wullink and Moura, \{Giovane C.M.\} and Moritz M{\"u}ller and Cristian Hesselman",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016 : Managing Everything toward a Secure, Smart, and Hyperconnected World, NOMS 2016 ; Conference date: 25-04-2016 Through 29-04-2016",

year = "2016",

month = jun,

day = "30",

doi = "10.1109/NOMS.2016.7502925",

language = "English",

pages = "913--918",

editor = "Badonnel, \{Sema Oktug\} and Mehmet Ulema and Cicek Cavdar and Granville, \{Lisandro Zambenedetti\} and \{dos Santos\}, \{Carlos Raniery P.\}",

booktitle = "Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium",

publisher = "IEEE",

address = "United States",

url = "https://noms2016.ieee-noms.org/, http://noms2016.ieee-noms.org/",

}

Wullink, M, Moura, GCM, Müller, M & Hesselman, C 2016, ENTRADA: A high-performance network traffic data streaming warehouse. in SO Badonnel, M Ulema, C Cavdar, LZ Granville & CRP dos Santos (eds), Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium., 7502925, IEEE, pp. 913-918, 2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016, Istanbul, Turkey, 25/04/16. https://doi.org/10.1109/NOMS.2016.7502925

ENTRADA: A high-performance network traffic data streaming warehouse. / Wullink, Maarten; Moura, Giovane C.M.; Müller, Moritz et al.
Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium. ed. / Sema Oktug Badonnel; Mehmet Ulema; Cicek Cavdar; Lisandro Zambenedetti Granville; Carlos Raniery P. dos Santos. IEEE, 2016. p. 913-918 7502925.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - ENTRADA

T2 - 2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016

AU - Wullink, Maarten

AU - Moura, Giovane C.M.

AU - Müller, Moritz

AU - Hesselman, Cristian

N1 - Conference code: 15

PY - 2016/6/30

Y1 - 2016/6/30

N2 - We present ENTRADA, a high-performance data streaming warehouse that enables researchers and operators to analyze vast amounts of network traffic and measurement data within interactive response times (seconds to few minutes), even in a small computer cluster. ENTRADA delivers such performance by employing a optimized file format and a high-performance query engine, both open-source. ENTRADA has been operational for more than 1.5 years, having ingested more than 100 TB of pcap files from two.nl DNS authoritative servers. As we discuss, we use this data in projects that aim at further increasing the security and stability of the.nl zone. We present in this paper our design choices, experiences, and a performance evaluation of ENTRADA. Finally, we open-source ENTRADA, which can be used out-of-the-box by researchers, operators, and registries to deploy their own networking analysis clusters for DNS traffic, and can be easily extended to handle any other structured data.

AB - We present ENTRADA, a high-performance data streaming warehouse that enables researchers and operators to analyze vast amounts of network traffic and measurement data within interactive response times (seconds to few minutes), even in a small computer cluster. ENTRADA delivers such performance by employing a optimized file format and a high-performance query engine, both open-source. ENTRADA has been operational for more than 1.5 years, having ingested more than 100 TB of pcap files from two.nl DNS authoritative servers. As we discuss, we use this data in projects that aim at further increasing the security and stability of the.nl zone. We present in this paper our design choices, experiences, and a performance evaluation of ENTRADA. Finally, we open-source ENTRADA, which can be used out-of-the-box by researchers, operators, and registries to deploy their own networking analysis clusters for DNS traffic, and can be easily extended to handle any other structured data.

KW - n/a OA procedure

UR - https://www.scopus.com/pages/publications/84979765599

U2 - 10.1109/NOMS.2016.7502925

DO - 10.1109/NOMS.2016.7502925

M3 - Conference contribution

AN - SCOPUS:84979765599

SP - 913

EP - 918

BT - Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium

A2 - Badonnel, Sema Oktug

A2 - Ulema, Mehmet

A2 - Cavdar, Cicek

A2 - Granville, Lisandro Zambenedetti

A2 - dos Santos, Carlos Raniery P.

PB - IEEE

Y2 - 25 April 2016 through 29 April 2016

ER -

ENTRADA: A high-performance network traffic data streaming warehouse

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this