FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n)

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.

Original languageEnglish
Title of host publication2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018
EditorsArslan Bromme, Andreas Uhl, Christoph Busch, Christian Rathgeb, Antitza Dantcheva
Place of PublicationPiscataway, NJ
PublisherIEEE
Number of pages5
ISBN (Electronic)978-3-88579-676-3
ISBN (Print)978-1-5386-6007-2
DOIs
Publication statusPublished - 10 Oct 2018
Event17th International Conference of the Biometrics Special Interest Group, BIOSIG 2018 - Darmstadt, Germany
Duration: 26 Sep 201828 Sep 2018
Conference number: 17

Publication series

NameInternational Conference of the Biometrics Special Interest Group (BIOSIG)
PublisherIEEE
Volume2018
ISSN (Print)1617-5468

Conference

Conference17th International Conference of the Biometrics Special Interest Group, BIOSIG 2018
Abbreviated titleBIOSIG 2018
CountryGermany
CityDarmstadt
Period26/09/1828/09/18

Fingerprint

confidence
intervals
lists
Ferromagnetic resonance
biometrics
scoring
ranking
Biometrics
Computational efficiency
costs
estimates
Costs

Keywords

  • Bootstrap Confidence Interval
  • Equal Error Rate
  • Open Source
  • Receiver operating characteristic

Cite this

Haasnoot, E., Khodabakhsh, A., Zeinstra, C., Spreeuwers, L., & Veldhuis, R. (2018). FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). In A. Bromme, A. Uhl, C. Busch, C. Rathgeb, & A. Dantcheva (Eds.), 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018 [8553607] (International Conference of the Biometrics Special Interest Group (BIOSIG); Vol. 2018). Piscataway, NJ: IEEE. https://doi.org/10.23919/BIOSIG.2018.8553607
Haasnoot, Erwin ; Khodabakhsh, Ali ; Zeinstra, Chris ; Spreeuwers, Luuk ; Veldhuis, Raymond. / FEERCI : A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018. editor / Arslan Bromme ; Andreas Uhl ; Christoph Busch ; Christian Rathgeb ; Antitza Dantcheva. Piscataway, NJ : IEEE, 2018. (International Conference of the Biometrics Special Interest Group (BIOSIG)).
@inproceedings{c188f1e4b1bd4247bd8045788944de74,
title = "FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n)",
abstract = "Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.",
keywords = "Bootstrap Confidence Interval, Equal Error Rate, Open Source, Receiver operating characteristic",
author = "Erwin Haasnoot and Ali Khodabakhsh and Chris Zeinstra and Luuk Spreeuwers and Raymond Veldhuis",
year = "2018",
month = "10",
day = "10",
doi = "10.23919/BIOSIG.2018.8553607",
language = "English",
isbn = "978-1-5386-6007-2",
series = "International Conference of the Biometrics Special Interest Group (BIOSIG)",
publisher = "IEEE",
editor = "Arslan Bromme and Andreas Uhl and Christoph Busch and Christian Rathgeb and Antitza Dantcheva",
booktitle = "2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018",
address = "United States",

}

Haasnoot, E, Khodabakhsh, A, Zeinstra, C, Spreeuwers, L & Veldhuis, R 2018, FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). in A Bromme, A Uhl, C Busch, C Rathgeb & A Dantcheva (eds), 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018., 8553607, International Conference of the Biometrics Special Interest Group (BIOSIG), vol. 2018, IEEE, Piscataway, NJ, 17th International Conference of the Biometrics Special Interest Group, BIOSIG 2018, Darmstadt, Germany, 26/09/18. https://doi.org/10.23919/BIOSIG.2018.8553607

FEERCI : A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). / Haasnoot, Erwin; Khodabakhsh, Ali; Zeinstra, Chris; Spreeuwers, Luuk; Veldhuis, Raymond.

2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018. ed. / Arslan Bromme; Andreas Uhl; Christoph Busch; Christian Rathgeb; Antitza Dantcheva. Piscataway, NJ : IEEE, 2018. 8553607 (International Conference of the Biometrics Special Interest Group (BIOSIG); Vol. 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - FEERCI

T2 - A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n)

AU - Haasnoot, Erwin

AU - Khodabakhsh, Ali

AU - Zeinstra, Chris

AU - Spreeuwers, Luuk

AU - Veldhuis, Raymond

PY - 2018/10/10

Y1 - 2018/10/10

N2 - Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.

AB - Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.

KW - Bootstrap Confidence Interval

KW - Equal Error Rate

KW - Open Source

KW - Receiver operating characteristic

UR - http://www.scopus.com/inward/record.url?scp=85060007289&partnerID=8YFLogxK

U2 - 10.23919/BIOSIG.2018.8553607

DO - 10.23919/BIOSIG.2018.8553607

M3 - Conference contribution

SN - 978-1-5386-6007-2

T3 - International Conference of the Biometrics Special Interest Group (BIOSIG)

BT - 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018

A2 - Bromme, Arslan

A2 - Uhl, Andreas

A2 - Busch, Christoph

A2 - Rathgeb, Christian

A2 - Dantcheva, Antitza

PB - IEEE

CY - Piscataway, NJ

ER -

Haasnoot E, Khodabakhsh A, Zeinstra C, Spreeuwers L, Veldhuis R. FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). In Bromme A, Uhl A, Busch C, Rathgeb C, Dantcheva A, editors, 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018. Piscataway, NJ: IEEE. 2018. 8553607. (International Conference of the Biometrics Special Interest Group (BIOSIG)). https://doi.org/10.23919/BIOSIG.2018.8553607